I'm very new to this so take what knowledge i have with a grain(or bucket!) of salt.
Cheapest and VERY safe: Armory. Using the offline storage method.
Best overall: Hardware solution i.e. Ledger nano S, Trezor, and the other one i can't remember

Downside to hardware solutions? BACKLOGGED ORDERS!!!! Ordered my nano s 6/15 and it isn't expected til early august. If you want to pay 4 times the cost, you can go on ebay or amazon and get them for 250+ USD(which is crazy, IMO)
Overall downsides to offline storage? Inability to rapidly buy/sell. Coinbase has been giving a large "fuck you", as you so well put lol, to the users lately and i suspect due to the nature of this game that it is intentional.
I just got in and put 6500 in btc and about 1300 in ether a few weeks ago.
I still have my btc on coinbase in hopes of catching the next wave.

Idk if i helped you at all or if you're reading this going "no shit!" but hey, it's what i know works for sure, right now.

Good luck my friend!

*As far as buying? I guess localbitcoins seems like the best option in the US. It's a bit more work but the fees are SO smaller.