A breed is categorized as a pony or a horse depending on it's body type, not it's height. Some breeds of ponies (poa's for example) are as tall as a horse.
Ponies have shorter legs in comparison to it's overall height, thicker necks and stockier, heavier bodies. They typically come from areas where weather is harsher and vegetation is less nutritious and terrain is more difficult.
"pony" can also be used as a term unrelated to the horses breed, as in "polo pony". Those are horses and not ponies at all.