Flying isn't extraordinarily difficult in and of itself, just as driving's not that difficult, left right stop go, just add a dimension Lol.
It's all of the knowledge, experience, and procedures that come with it. There's a lot of memorization, repetition to build good habits and muscle memory. Situational awareness and judgement are also big factors, knowing when or when not to fly, knowing where you are & who/what's around you.
But the bare bones stick and rudder stuff is pretty simple. If you've never flown in a small airplane I highly recommend searching for a local flight school and seeing if they do introductory flights, that's how I got into it.