Ph.D. in mathematics. Getting a Ph.D. is difficult for anyone. If you succeed of not has to do with many factors, including your health, how well you tolerate stress and criticism, how well you get along with your advisor, how smoothly your research goes, if you have a supportive social situation, but mostly how much you really want to get it despite all the setbacks you will have. I know very few people who thought graduate school was "fun", although I do know a few where all the stars aligned and they had for the most part smooth sailing.
For me it was a dream that I wanted to achieve, and when I failed, I picked my ass up and tried again. I'd like to say that that was fortitude or commitment, but really it was my ego that couldn't tolerate the failure.
Was it worth it in the end? In the greater context, probably not. I'm sure I could have found something else to do that I would have enjoyed with a LOT less stress, and focused more on building a successful career or family life.
Would I advise people to get a Ph.D. and try for academic post? Definitely not. Unless you're extremely focused and career oriented...almost more interested in your career as a scientist than doing the science itself...you are almost surely not going to get good job. The competition is beyond extreme.
If you want to make less money than a janitor, have a 3/4-time benefitless semester-to-semester job, and teach dumb-asses high school level material all day long...well then...the community colleges are always hiring fresh Ph.D.s