A short thread on Playwright (github.com/microsoft/play), the new library to instrument browsers. (1/N)
Conversation
~2 years ago, Puppeteer was released. We went from headless browsers, such as PhantomJS, that could be easily detected using their fingerprint, to Headless Chrome (HC), that had a fingerprint almost like the one of a normal Chrome. (2/N)
1
1
Despite these differences, it was still possible to detect headless Chrome due to more (or less) subtle inconsistencies: lack of plugins, presence of navigator.webdriver, inconsistent permissions for notifications, etc. (antoinevastel.com/bot%20detectio, antoinevastel.com/bot%20detectio) (3/N)
1
2
With Playwright, the new instrumentation framework of Microsoft, the game goes on. Playwright can be used to instrument (Headless) Chrome/Chromium, (Headless) Firefox and Webkit using the same API, which removes the burden of developing multiple programs for each browser. (4/N)
1
1
Thus, there will certainly be a rise of bots based on this framework in the coming weeks. Although the detection of HCs instrumented with Playwright is similar to the ones instrumented with Pptr, detecting instrumented Firefox and Webkit browsers will be an interesting challenge.
1
1
The first analysis of their fingerprint shows that:
- By default, Firefox (with Playwright) has a genuine user-agent + navigator.webdriver = false (like a normal Firefox browser)
(6/N)
1
1
- Webkit (~Safari) has an identifiable user-agent but navigator.webdriver = false (like a normal Safari). Thus, once its user agent has been modified, it becomes less easily identifiable. (7/N)
1
1
Although these browsers may not be too easily too identifiable, it does not mean they can't be caught using either existing techniques, new JavaScript challenges, and ML behavioral techniques. (8/N)
1
1
However, contrary to when I was in Ph.D., I won't release these new techniques to avoid making my job of bot hunter* (ok, *dev) more difficult, but I'm pretty sure it's a matter of time before new cat and mouse repositories will appear for browsers instrumented with Playwright.