Case Study: Parsing 1 Million Product Pages Per Day Using Pineapple Proxy + Playwright

The Challenge

A large e-commerce analytics platform needed to collect pricing data, product descriptions, availability status, and customer reviews from 200+ online retailers across 30 countries. Their target: 1 million product pages per day.

The Constraints

Why Playwright?

Traditional HTTP-based scraping (requests, Scrapy) failed against JavaScript-rendered content and advanced bot detection. Playwright provides:

• Full browser rendering (Chromium, Firefox, WebKit)
• JavaScript execution for dynamic content
• Realistic browser fingerprints
• Stealth plugin support (evades detection)

The Architecture

Proxy Manager (Pineapple API) ↓ Dispatcher (Celery + Redis) ↓ Worker Pool (200 Playwright instances) ↓ Each Worker: 1. Get proxy from pool 2. Launch browser with proxy 3. Navigate to product URL 4. Wait for content 5. Extract structured data 6. Submit to processing pipeline 7. Rotate proxy ↓ Data Processing Pipeline ↓ Analytics & Storage

Proxy Infrastructure

We deployed 5,000 residential proxies from Pineapple Proxy, distributed across 30 target countries. Key configuration:

# Proxy assignment strategy
proxy_pool = {
    "US": {"count": 1500, "type": "residential", "rotation": "per-request"},
    "UK": {"count": 500, "type": "residential", "rotation": "per-request"},
    "DE": {"count": 400, "type": "residential", "rotation": "per-request"},
    "JP": {"count": 300, "type": "residential", "rotation": "per-request"},
    # ... 26 more countries
}

Playwright Stealth Configuration

from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync

def create_browser(proxy_url: str):
    playwright = sync_playwright().start()
    browser = playwright.chromium.launch(
        headless=True,
        proxy={"server": proxy_url}
    )
    context = browser.new_context(
        viewport={"width": 1920, "height": 1080},
        user_agent="Mozilla/5.0 ...",
        locale="en-US",
        timezone_id="America/New_York"
    )
    page = context.new_page()
    stealth_sync(page)  # Anti-detection
    return browser, page

Handling Anti-Bot Challenges

Cloudflare Challenge

def handle_cloudflare(page):
    try:
        # Wait for challenge to resolve
        page.wait_for_selector(
            "cf-turnstile", state="attached", timeout=5000
        )
        # Wait for actual content
        page.wait_for_load_state("networkidle")
        return True
    except:
        return False

Rate Limiting Detection

def safe_request(page, url, max_retries=5):
    for attempt in range(max_retries):
        try:
            response = page.goto(url, timeout=30000)
            if response.status == 200:
                return page.content()
            elif response.status == 429:
                # Rate limited — rotate proxy
                rotate_proxy()
                time.sleep(2 ** attempt)  # Exponential backoff
            else:
                return None
        except:
            rotate_proxy()
            time.sleep(5)
    return None

Results

Key Lessons

What Worked

1. Residential proxies were essential — datacenter proxies had 30%+ block rates
2. Geo-targeted pools reduced Cloudflare challenges by 40%
3. Per-request rotation prevented IP-based fingerprinting
4. Playwright stealth dramatically reduced detection rates

What Didn't

1. Sticky sessions caused more blocks (sites tracked session behavior)
2. Chrome-only occasionally detected; mixing Firefox helped
3. Too-fast rotation triggered behavioral detection

Cost Analysis

Proxy infrastructure: $X,XXX/mo Compute (200 workers): $X,XXX/mo Bandwidth: $XXX/mo Maintenance: $XXX/mo ----------------------------------- Total: ~$X.XX per 1K pages

Conclusion

Scaling to 1M pages per day requires robust proxy infrastructure, intelligent browser automation, and constant adaptation to anti-bot measures. Pineapple Proxy's residential pool provided the reliability and global coverage needed.

Ready to scale your scraping operation? Check our enterprise plans or browse our proxy inventory.