Happy 420 Day! Not sure which explanation I like most but the one I glom onto deals with some California high schoolers who were serving detention and agreed that after their one-hour punishment they would meet in the woods behind the school at 4:20 in the afternoon to toke up.
Here are several stories about rock stars and their run-ins with the 'sweet leaf' (thanks Ozzy)
Btw, I never inhaled.
I've also included one of the greatest Key And Peele bits ever!