The Challenge

A large e-commerce analytics platform needed to collect pricing data, product descriptions, availability status, and customer reviews from 200+ online retailers across 30 countries. Their target: 1 million product pages per day.

The Constraints

Constraint	Detail
Pages per day	1,000,000
Target sites	200+ e-commerce platforms
Countries	30
Anti-bot measures	Cloudflare, DataDome, Akamai, custom JS challenges
Data freshness	Max 24 hours between updates
Budget	$X,XXX/month for proxy infrastructure

Why Playwright?

Traditional HTTP-based scraping (requests, Scrapy) failed against JavaScript-rendered content and advanced bot detection. Playwright provides:

Full browser rendering (Chromium, Firefox, WebKit)
JavaScript execution for dynamic content
Realistic browser fingerprints
Stealth plugin support (evades detection)

The Architecture

Proxy Manager (Pineapple API)
       ↓
Dispatcher (Celery + Redis)
       ↓
Worker Pool (200 Playwright instances)
       ↓
  Each Worker:
    1. Get proxy from pool
    2. Launch browser with proxy
    3. Navigate to product URL
    4. Wait for content
    5. Extract structured data
    6. Submit to processing pipeline
    7. Rotate proxy
       ↓
Data Processing Pipeline
       ↓
Analytics & Storage

Proxy Infrastructure

We deployed 5,000 residential proxies from Pineapple Proxy, distributed across 30 target countries. Key configuration:

# Proxy assignment strategy
proxy_pool = {
    "US": {"count": 1500, "type": "residential", "rotation": "per-request"},
    "UK": {"count": 500, "type": "residential", "rotation": "per-request"},
    "DE": {"count": 400, "type": "residential", "rotation": "per-request"},
    "JP": {"count": 300, "type": "residential", "rotation": "per-request"},
    # ... 26 more countries
}

Playwright Stealth Configuration

from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync

def create_browser(proxy_url: str):
    playwright = sync_playwright().start()
    browser = playwright.chromium.launch(
        headless=True,
        proxy={"server": proxy_url}
    )
    context = browser.new_context(
        viewport={"width": 1920, "height": 1080},
        user_agent="Mozilla/5.0 ...",
        locale="en-US",
        timezone_id="America/New_York"
    )
    page = context.new_page()
    stealth_sync(page)  # Anti-detection
    return browser, page

Handling Anti-Bot Challenges

Cloudflare Challenge

def handle_cloudflare(page):
    try:
        # Wait for challenge to resolve
        page.wait_for_selector(
            "cf-turnstile", state="attached", timeout=5000
        )
        # Wait for actual content
        page.wait_for_load_state("networkidle")
        return True
    except:
        return False

Rate Limiting Detection

def safe_request(page, url, max_retries=5):
    for attempt in range(max_retries):
        try:
            response = page.goto(url, timeout=30000)
            if response.status == 200:
                return page.content()
            elif response.status == 429:
                # Rate limited — rotate proxy
                rotate_proxy()
                time.sleep(2 ** attempt)  # Exponential backoff
            else:
                return None
        except:
            rotate_proxy()
            time.sleep(5)
    return None

Results

Metric	Target	Achieved
Daily pages scraped	1,000,000	1,023,000
Success rate	>95%	97.3%
Average page load	<5s	3.2s
Proxy block rate	<5%	2.1%
Data accuracy	>99%	99.4%
Cost per 1K pages	—	$0.42

Key Lessons

What Worked

Residential proxies were essential — datacenter proxies had 30%+ block rates
Geo-targeted pools reduced Cloudflare challenges by 40%
Per-request rotation prevented IP-based fingerprinting
Playwright stealth dramatically reduced detection rates

What Didn't

Sticky sessions caused more blocks (sites tracked session behavior)
Chrome-only occasionally detected; mixing Firefox helped
Too-fast rotation triggered behavioral detection

Cost Analysis

Proxy infrastructure:     $X,XXX/mo
Compute (200 workers):    $X,XXX/mo
Bandwidth:                $XXX/mo
Maintenance:              $XXX/mo
-----------------------------------
Total:                    ~$X.XX per 1K pages

Conclusion

Scaling to 1M pages per day requires robust proxy infrastructure, intelligent browser automation, and constant adaptation to anti-bot measures. Pineapple Proxy's residential pool provided the reliability and global coverage needed.

Ready to scale your scraping operation? Check our enterprise plans or browse our proxy inventory.