I loved your question, because I realized I've always wondered too, and I watch BBC a lot. So I looked it up. :) The best guess is it started in the early 1800's with Anna, 7th Duchess of Bedford. Evening meal wasn't served until around 8-9pm. Midway between lunch and dinner, she said she got a "sinking feeling" (what we'd call the "3-o'clock slump" these days). So she started having tea, bread&butter, and little cakes served at around 3-4PM in her bedroom, while she was receiving her female visitors. The idea caught on big and was eventually called Afternoon Tea. We just call it an afternoon coffee break in the US and its not nearly so fancy. Ah well.
It became a customer among the rich in the 1840's. It was considered a small meal, like an afternoon Brunch. Slowly it worked its way into everyday life and by the end of the 1800's it was observed by both the rich and the middle class. It is called low tea. Low tea is observed between 4 and 6 pm. As time passed it became more of a social observance than an actual afternoon meal. There is also High tea which is observed between 5 - 7 pm by the lower class as their evening meal.