Do exhaustion levels affect Spellcasting abilities in any specific way?

Are spellcasters only affected under the Level 3 generic condition for disavantages and... that's it?

Shouldn't exhaustion levels affect spellcasting a bit more? If not, why?

Is this another martial x caster bias?