They have a lot of people to attend to. And it's not strictly first come - first served like getting a haircut, they take the most urgent cases first. You could be sitting in the waiting room with a broken ankle. If they rush a guy in with a sucking chest wound he's going to be seen ahead of you. The best bet is, unless it truly is an emergency, make an appointment with your primary doctor or visit a walk-in clinic.
Because they get saturated with people who do not need it's services but use it as a doctors office visit. If true ER cases were the only patients seen, you would see the time to be seen drop drastically.