What's the most advanced math course/topic?
It depend on the curriculum. Personally I'd say that calculus is the hardest and most complicated, but different curriculums have different levels of difficulty and different topics.
At my high school, the most difficult it gets is AP Calculus D and Linear Algebra. So I'm going to go with calc.