Maybe a dumb question, but is there a physiological reason draft horses aren't ridden more?

I mean, other than them being fuckass big. Is there anything in their muscular/skeletal systems that means they're better at pulling than carrying? I get why you wouldn't want a small riding horse pulling a heavy load, but is there any reason not to put a rider on a draft horse aside from maybe the physical comfort of the rider?

Edit: Well after reading all your responses I now really, really want to ride a draft horse.