"AI will give everyone access to a genius" isn't the flex they think it is
I was once asked by a lecturer to write a short program for him. He explained what he wanted it to do and explained how he wanted me to code it.
It was a randomised array generator. He wanted to give it a number of variables, I forget how many, and it was to output 200 rows of those variables in a random order. Obviously with no repeats.
So:
1 2 3 4 5 2 3 4 1 5 3 4 1 5 2 etc...
He said I should just have it create the 200 lines at random, check for duplicates and repeat until there were no duplicates. I did not do that. I programmed it properly and then spent just as long as it took me code hanging out with a colleague who was better at stats than me. Between us we calculated how long it would take for the program to finish if we did it his way.
If all the computers on the planet worked together, it would still take until the heat death of the universe to randomly come across a list that long with no duplicates.
The guy was a Professor. This is in the UK, where that is an earned title, not just a lecturer. It didn't matter to him that his PhD was in an unrelated field.
It really doesn't matter how smart the person you're giving orders to is if those orders are stupid. I often engage in malicious compliance, which has the same effect as an AI that won't say no. People ask for things that can't be done, won't work or will make things worse and they fuck everything up. These are exactly the kind of people that love AI.
They don't know what that's like because they've always been the ones asking for stupid things.