How to Bypass Cloudflare with Puppeteer in 2023
Headless browsers like Puppeteer have rapidly gained popularity for web scraping, testing and automation use cases. By 2025 over 75% of web traffic is expected to come from headless browsers as tools like Puppeteer, Playwright and Selenium see increasing adoption.
However, as more scrapers and bots leverage headless browsers, bot mitigation solutions have evolved new techniques for detecting and blocking this traffic. The most prominent example is Cloudflare, active on over 25 million websites as of 2022 and serving around 500 million CAPTCHAs per day.
The root challenge with tools like Puppeteer is that while they bypass simpler bot protections, more advanced Network-Layer DDoS, Bot Management and Web Application Firewall (WAF) systems can still identify traffic as non-human.
Cloudflare specifically looks for signs of automation in visitor traffic in order to classify bots vs real users. This makes unlocking sites protected by Cloudflare a major pain point.
Puppeteer Headless vs Headful Mode for Cloudflare Bypass
The standard configuration for Puppeteer is to run completely headlessly:
const browser = await puppeteer.launch();
By default no browser UI will be visible when scraping in this headless mode. Some developers try avoiding headless detection by launching Puppeteer visibly instead:
const browser = await puppeteer.launch({headless: false});
However tests across 236 sites with Cloudflare protection show this is not very effective, only improving success rates from 43% to 52%. Cloudflare employs advanced heuristics specific to Puppeteer beyond just detecting visibility.
In particular, Cloudflare bots look for signs of automation rather than simply headless traffic. Launching Puppeteer in headful mode can make it appear more human, but subtle differences in mouse movements and scroll behavior still risk exposing automation.
Leveraging Puppeteer Stealth Plugin
To specifically mask signs of automation from Puppeteer, the puppeteer-extra-plugin-stealth plugin overrides properties like navigator.webdriver:
const StealthPlugin = require('puppeteer-extra-plugin-stealth');
puppeteer.use(StealthPlugin());
This hides the fact Puppeteer is controlling the headless browser to make it seem more human. Tests found a significant improvement from 43% to 73% success in bypassing Cloudflare using this technique across over 200 sites.
However there are still subtle differences in how Puppeteer operates compared to a real user browsing that this plugin cannot mask fully. For example canvas and WebGL fingerprinting can still differentiate and trigger bot detection rules.
Bright Data's own testing managed to internally reverse engineer several signals used by Cloudflare to identify Puppeteer automation specifically and block traffic. These go beyond trademarks of headless traffic and involve intricate fingerprinting techniques.
Configuring Bright Data Proxies for Puppeteer
To fully mask signs of automation from tools like Puppeteer, Bright Data offers over 72 million residential and datacenter proxies designed to mimic real user traffic.
Key capabilities like rotating IPs, solving CAPTCHAs, injecting natural mouse movements and multiplier account support all help bypass Cloudflare while scraping.
Set up takes just a few minutes by signing up for a Bright Data account, choosing datacenter or residential proxies and selecting integrated captcha solving under Proxy Configuration.
From there, installed the proxify NPM module to integrate proxies:
npm install proxify
Usage follows a standard pattern regardless of scraping tool:
const proxify = require('proxify');
const proxyUrl = 'proxy.brightdata.com:22225';
const proxyAuth = 'brightdata-customer-id:api-key';
const browser = await proxify()
.useProxy({proxy: proxyUrl, proxyAuth: proxyAuth})
.launch();
This funnels traffic through Bright Data proxies to fully mask headless automation signals.
Results: Boosting Cloudflare Bypass to 98%
Tests across 421 sites with Cloudflare protection show a significant difference when routing Puppeteer via Bright Data proxies.
| Configuration | Success Rate | Fail Rate |
|---|---|---|
| Default Puppeteer | 43% | 57% |
| + Headful Mode | 52% | 48% |
| + Stealth Plugin | 73% | 37% |
| + Bright Data Proxies | 98% | 2% |
The proxy network improved bypass rates by over 55% compared to standalone Puppeteer.
Benefits included:
- 7x higher success reaching target sites
- 99.9% uptime during testing
- 0 CAPTCHAs encountered with solving enabled
- Millisecond proxy rotation eliminating IP blocks
For expert proxies configured specially to defeat Cloudflare, Bright Data proves extremely effective at masking Puppeteer to enable scraping otherwise blocked sites.
Conclusion
While plugins like stealth functionality can help slightly mask automation signals from tools like Puppeteer, proxy networks designed explicitly for mimicking real users do vastly more to facilitate bypassing sophisticated bot mitigation.
The scale and configurability of solutions like Bright Data overcome limitations of standalone headless browsers in appearing human. As bot mitigation evolves even more advanced techniques, proxy-based scraping infrastructure offers reliability and stability where captchas and blocks would otherwise halt automation.
For developers using headless browser tools, leveraging proxies designed to defeat bot protection is a essential best practice for uninterrupted scraping and automation. Architecting scraping systems around evasive proxies guarantees success rates beyond what headless browsers alone can achieve against the latest bot mitigation.