Computer Vision + mouse and keyboard scraper?

The typical approach to a web scraper is to manipulate the website’s api or its front end elements to obtain the data required. When possible, this is preferable to messing with automated browsers because data can be obtained much quicker and more directly this way, but it can also be more sensitive to anti-bot detection. The difference between how a human will naturally scroll a website versus a scraper manipulating requests and drilling for the data it needs tends to stand out, and much time has been spent by both scrapers and web designers in the arms race of bot and anti-bot detection.

Maybe, as AI tools get better and cheaper while bot detection improves, there will come a time where its more efficient to make a scraper that uses computer vision to understand the webpage visually and then sends commands to manipulate the mouse and keyboard accordingly. This project is more difficult and slower than the typical requests scraper, but it also interfaces with the website in a way much more similar to how a real human, so it’s potentially stronger against some forms of behavioral detection.

The other big problem with this kind of scraper is scalability. You cannot use headless browsers, and if you go as far as mouse and keyboard manipulation than you either need a virtual machine or it will totally occupy your pc while running. It might still be a good choice when you don’t need a high quantity of data scraped but its very important or difficult to evade anti bot measures.

Has anyone messed around with the idea of making a scraper like this?