How We Validate 50,000+ Proxies per Day: Inside Pineapple Proxy's Architecture

The Validation Challenge

The internet is full of proxy lists. Most are outdated, inaccurate, or outright useless. At Pineapple Proxy, we process 50,000+ proxy endpoints daily to deliver verified, working proxies to our users.

This is how we do it.

Pipeline Overview

┌──────────────────┐ GitHub Repos ─────▶│ Harvester │ Community Lists ──▶│ (Collector) │────▶ RabbitMQ Exchange └──────────────────┘ │ (fanout) ▼ ┌──────────────────┐ │ Validator │ │ (Multi-protocol │ │ testing) │ └──────────────────┘ │ ▼ ┌──────────────────┐ │ Enricher │ │ (Geo, ISP, │ │ anonymity) │ └──────────────────┘ │ ▼ ┌──────────────────┐ │ Database + API │ │ (Served to you) │ └──────────────────┘

Stage 1: Harvesting

Our harvesters continuously collect proxy candidates from:

• 50+ GitHub repositories (updated hourly)
• Public proxy lists and forums
• Community submissions
• Our own scanning infrastructure

Each candidate is published as an event to RabbitMQ:

public class ProxyDiscoveredEvent
{
    public string Ip { get; set; }
    public int Port { get; set; }
    public string Source { get; set; }
    public DateTime DiscoveredAt { get; set; }
}

We use RabbitMQ fanout exchanges to broadcast discoveries to multiple validator instances.

Stage 2: Validation

The validator is a .NET 10 service running on MassTransit. It performs real-world testing:

public class ProxyValidatorConsumer : IConsumer<ProxyDiscoveredEvent>
{
    private readonly IHttpClientFactory _httpFactory;

    public async Task Consume(ConsumeContext<ProxyDiscoveredEvent> context)
    {
        var message = context.Message;
        var tasks = new[]
        {
            TestHttp(message.Ip, message.Port),
            TestHttps(message.Ip, message.Port),
            TestSocks5(message.Ip, message.Port),
        };

        var results = await Task.WhenAll(tasks);
        var proxyData = Aggregate(results, message);
        await Save(proxyData);
    }
}

Protocol Tests

HTTP Test: Send a real HTTP request through the proxy to our verification endpoint.

async Task<HttpResult> TestHttp(string ip, int port)
{
    var handler = new HttpClientHandler
    {
        Proxy = new WebProxy($"http://{ip}:{port}")
    };
    using var client = new HttpClient(handler);
    client.Timeout = TimeSpan.FromSeconds(10);

    try
    {
        var response = await client.GetAsync("https://httpbin.org/ip");
        var body = await response.Content.ReadAsStringAsync();
        return new HttpResult(response.IsSuccessStatusCode, body);
    }
    catch
    {
        return new HttpResult(false, null);
    }
}

HTTPS Test: Verify CONNECT tunnel establishment.

SOCKS5 Test: Full SOCKS5 negotiation including UDP support check.

Stage 3: Anonymity Verification

This is where most aggregators stop and we keep going. We inspect response headers:

public AnonymityLevel DetermineAnonymity(HttpResponseHeaders headers)
{
    var hasXForwardedFor = headers.Contains("X-Forwarded-For");
    var hasVia = headers.Contains("Via");
    var hasXRealIp = headers.Contains("X-Real-IP");
    var hasClientIp = headers.Contains("Client-IP");

    if (!hasXForwardedFor && !hasVia && !hasXRealIp && !hasClientIp)
        return AnonymityLevel.Elite;

    if (hasXForwardedFor || hasXRealIp || hasClientIp)
    {
        var xff = headers.GetValues("X-ForwardedFor").FirstOrDefault();
        if (xff != null && xff != proxyIp)
            return AnonymityLevel.Transparent; // Leaks real IP
    }

    return AnonymityLevel.Anonymous;
}

Stage 4: Geo-Enrichment

Each validated proxy is enriched with:

• Country and city (verified, not just DB lookup)
• ISP information
• Response time percentiles
• Last verification timestamp

Stage 5: Storage and Serving

Validated proxies are stored in our database and served via REST API. The public endpoint at GET /api/ProxyDemo/Index returns:

{
  "discovered": {
    "firstDiscoveryUTC": "2026-01-15T10:30:00Z",
    "lastDiscoveryUTC": "2026-03-28T14:22:00Z",
    "proxyPoint": {
      "proxy": "123.45.67.89:8080",
      "protocol": "http",
      "ip": "123.45.67.89",
      "port": 8080,
      "anonymity": "elite",
      "score": 0.94,
      "geolocation": {
        "Country": "United States",
        "City": "New York"
      }
    }
  },
  "verifiedOnUTC": "2026-03-28T14:20:00Z",
  "verifiedHttpOS": true,
  "verifiedHttpBrowser": true,
  "verifiedCity": "New York",
  "verifiedCountry": "United States"
}

Scaling

Our validator runs across multiple instances. RabbitMQ's fanout exchange ensures every discovery reaches every validator — enabling parallel testing without duplication.

Harvester → [RabbitMQ Fanout] → Validator 1 → Validator 2 → Validator N (auto-scale)

Validation Dashboard

We track real-time metrics:

Proxies tested today: 52,341 Average test time: 3.2s Success rate: 32% Active proxies in DB: 18,442 Last full cycle: 2 min ago

What We Test That Others Don't

Conclusion

Our validation pipeline runs 24/7, processing 50K+ proxy endpoints daily through multi-protocol testing, header inspection, and geo-verification. The result: a proxy list you can actually trust.

See the live results on our Proxy List page or explore the Statistics dashboard.