Web Scraping Without Getting Blocked: Proxy Rotation, User-Agents, and Headers

The Arms Race of Web Scraping

Websites are spending billions on anti-bot technology. Cloudflare, DataDome, Akamai, and PerimeterX are deployed across most major sites. If you're scraping without anti-detection measures, you're leaving money on the table.

This guide covers the techniques that actually work in 2026.

Why Scrapers Get Blocked

Sites detect scrapers through multiple signals:

Detection Signal Weight ───────────────────────────────────── IP behavior (rate, volume) 40% Browser fingerprint 30% Request headers 15% JavaScript challenges 10% Behavioral patterns 5%

No single signal gets you blocked. It's the combination.

Strategy 1: Proxy Rotation

Basic Rotation

import random
import requests

proxies = [
    "http://proxy1:8080",
    "http://proxy2:8080",
    "http://proxy3:8080",
]

def fetch(url):
    proxy = random.choice(proxies)
    try:
        return requests.get(url, proxies={"http": proxy, "https": proxy})
    except:
        proxies.remove(proxy)  # Remove dead proxies
        return fetch(url)      # Retry with another

Problem: Sites detect patterns in rotation. Random is predictable.

Smart Rotation

class SmartRotator:
    def __init__(self, proxy_pool):
        self.pool = proxy_pool
        self.usage = {}  # Track requests per proxy

    def get_proxy(self, url):
        """Select best proxy for a specific URL."""
        domain = urlparse(url).netloc

        # Avoid using same proxy for same domain too often
        available = [
            p for p in self.pool
            if self.usage.get(p, {}).get(domain, 0) < 3
        ]

        if not available:
            available = self.pool
            self.usage.clear()  # Reset counters

        proxy = random.choice(available)

        # Track usage
        self.usage.setdefault(proxy, {})
        self.usage[proxy][domain] = self.usage[proxy].get(domain, 0) + 1

        return proxy

Geo-Targeted Rotation

Match proxy geography to target:

def get_proxy_for_target(target_url):
    """Select proxy in same region as target server."""
    target_country = geoip(target_url)
    region_proxies = pool.get_proxies_by_country(target_country)

    if region_proxies:
        return random.choice(region_proxies)
    return random.choice(pool.all)  # Fallback

Strategy 2: Browser Fingerprinting

HTTP clients (requests, Scrapy) have obvious fingerprints. Modern scraping requires full browsers.

Playwright with Stealth

from playwright.sync_api import sync_playwright
from playwright_stealth import stealth_sync

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()

    # Apply stealth patches
    stealth_sync(page)

    # This makes headless Chrome look like a real Chrome
    page.goto("https://bot.sannysoft.com")

    # Without stealth: 14/17 detection tests FAIL
    # With stealth:     2/17 detection tests FAIL

What Stealth Modifies

• WebDriver flag (removed)
• Chrome runtime (masked)
• Permissions (normalized)
• WebGL fingerprint (realistic)
• Canvas noise (added)
• Font list (realistic)

Randomizing Viewport

def random_viewport(page):
    sizes = [
        {"width": 1920, "height": 1080},
        {"width": 1366, "height": 768},
        {"width": 1536, "height": 864},
        {"width": 1440, "height": 900},
    ]
    size = random.choice(sizes)
    page.set_viewport_size(size)

Strategy 3: Request Headers

Real browsers send consistent, ordered headers:

BASE_HEADERS = {
    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    "Accept-Language": "en-US,en;q=0.5",
    "Accept-Encoding": "gzip, deflate, br",
    "Connection": "keep-alive",
    "Upgrade-Insecure-Requests": "1",
    "Sec-Fetch-Dest": "document",
    "Sec-Fetch-Mode": "navigate",
    "Sec-Fetch-Site": "none",
    "Sec-Fetch-User": "?1",
}

Common Mistakes

• Missing Accept-Encoding — Bots often omit this
• Wrong order — Browsers have specific header ordering
• Consistent headers — Always identical is suspicious
• Missing Sec- headers* — Modern browsers all send these

Strategy 4: Human Mimicry

Random Delays

import time
import random

def human_delay():
    """Wait like a human would."""
    delay = random.gauss(3, 1)  # Normal distribution
    delay = max(1, min(10, delay))  # Clamp
    time.sleep(delay)

Mouse Movements

async def human_like_navigation(page, url):
    await page.goto(url)

    # Scroll like a human
    for _ in range(random.randint(2, 5)):
        await page.mouse.wheel(0, random.randint(200, 600))
        await page.wait_for_timeout(random.randint(500, 2000))

    # Move mouse randomly
    for _ in range(random.randint(3, 8)):
        x = random.randint(100, 1800)
        y = random.randint(100, 900)
        await page.mouse.move(x, y, steps=10)
        await page.wait_for_timeout(random.randint(100, 300))

Strategy 5: Captcha Handling

When captchas do appear:

class CaptchaHandler:
    def __init__(self):
        self.captcha_count = 0

    def handle(self, page):
        """Detect and respond to captchas."""
        # Option 1: Rotate proxy and retry
        if self.captcha_count < 3:
            self.captcha_count += 1
            return "rotate_proxy"

        # Option 2: Use solving service
        # captcha_solver = CapSolver(api_key)
        # return captcha_solver.solve(page)

        # Option 3: Slow down dramatically
        return "slow_down"

Putting It All Together

class AntiDetectionScraper:
    def __init__(self, proxy_pool):
        self.rotator = SmartRotator(proxy_pool)
        self.captcha = CaptchaHandler()

    def scrape(self, url):
        proxy = self.rotator.get_proxy(url)
        browser = self._create_browser(proxy)

        try:
            page = browser.new_page()
            stealth_sync(page)
            random_viewport(page)

            human_delay()
            response = page.goto(url, wait_until="networkidle")

            if "captcha" in response.url.lower():
                action = self.captcha.handle(page)
                if action == "rotate_proxy":
                    browser.close()
                    return self.scrape(url)  # Retry with new proxy

            human_like_navigation(page)
            return page.content()

        finally:
            browser.close()

Recommended Tool Stack

Testing Your Setup

Use these sites to verify your anti-detection:

• https://bot.sannysoft.com — Browser fingerprint tests
• https://amiunique.org — Browser uniqueness
• https://iphey.com — Bot detection score
• https://pixelscan.net — Bot detection analysis

Conclusion

Getting blocked is not a question of if, but when — unless you implement proper anti-detection. Proxy rotation, browser fingerprinting, header management, and human mimicry together create a scraper that's indistinguishable from a real user.

Pineapple Proxy provides the residential proxy infrastructure to power your scraping operation. Browse our proxies or see our plans.