Those who argue that math and physics is, like, the hardest, have probably never really studied any other field. I have no talent for math, but I know that means I just have no talent, not that it has any particular claim for supreme difficulty.
For instance, here is the a bit from a book on linguistics I am reading, The Syntax of Comparative Constructions Operators, Ellipsis Phenomena and Functional Left Peripheries by Julia Bacskai-Atkari:

I do have talent for this area (unlike math) and I enjoy it, so that is pretty comprehensible to me, despite being at a post-grad level. But maybe one out of 100,000 people would have any clue what that is talking about, and no math person would.
But sure, the only difficult (โhardโ) fields with any rigor are math and physics. Right. (In reality, it just means that those are the only fields social fuck-ups can easily prosper in.)