Switching proxies in .NET without creating a new HttpClient every time

I was working on a scraper and ran into something that sounded simple at first: use a different proxy for each request.

Not just a different port on the same host, but sometimes a completely different proxy server. The proxy list could eventually come from a database. Some proxies needed to be disabled temporarily when they started failing. And the scraper still had to keep running.

My first thought was: why not just change the proxy before the next request?

handler.Proxy = new WebProxy("http://proxy-a.example:8080");
await client.GetAsync(url);

handler.Proxy = new WebProxy("http://proxy-b.example:8080");
await client.GetAsync(url);

That does not work. Once a handler has been used, you cannot change properties like this anymore. You get errors like:

InvalidOperationException: This instance has already started one or more requests.
Properties can only be modified before sending the first request.

That makes sense. The handler owns connection pools and internal state. Changing that while requests are already going through it would get messy.

The clunky option

The next option is obvious too: create one HttpClient per proxy.

In ASP.NET, that usually turns into a bunch of named clients:

services.AddHttpClient("proxy-1")
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        Proxy = new WebProxy("http://proxy-1.example:8080"),
        UseProxy = true
    });

services.AddHttpClient("proxy-2")
    .ConfigurePrimaryHttpMessageHandler(() => new SocketsHttpHandler
    {
        Proxy = new WebProxy("http://proxy-2.example:8080"),
        UseProxy = true
    });

For a couple of fixed proxies, that is fine. For scraping, it quickly feels wrong.

In that kind of system, proxies are usually not static configuration. They are data. You fetch them from a database, track which ones fail, temporarily disable bad ones, add new ones, remove old ones. You do not want a separate client registration for every possible proxy.

What I wanted was this:

one reusable HttpClient
one handler that does not need to be recreated all the time
choose a proxy per request
load proxies from a pool or database
no named client per proxy

.NET already has an interface for this.

IWebProxy

SocketsHttpHandler.Proxy does not have to be a plain WebProxy. You can give it your own implementation of IWebProxy.

The interface is small:

Uri? GetProxy(Uri destination);

That is where the solution sits. The handler stays the same, but when a request is made, .NET asks your proxy object which proxy to use.

So you do not keep changing the handler. You let GetProxy return a different answer.

Simple implementation

A minimal version looks like this:

public sealed class RotatingProxy : IWebProxy
{
    private readonly IProxyProvider _provider;

    public RotatingProxy(IProxyProvider provider)
    {
        _provider = provider;
    }

    public ICredentials? Credentials { get; set; }

    public Uri? GetProxy(Uri destination)
    {
        return _provider.GetNextProxy(destination);
    }

    public bool IsBypassed(Uri host)
    {
        return false;
    }
}

The provider can get the proxy from wherever you want. It can be an in-memory list, or a database-backed pool with health checks.

For example, round-robin:

public interface IProxyProvider
{
    Uri? GetNextProxy(Uri destination);
}

public sealed class RoundRobinProxyProvider : IProxyProvider
{
    private readonly object _lock = new();
    private readonly IReadOnlyList<Uri> _proxies;
    private int _index;

    public RoundRobinProxyProvider(IReadOnlyList<Uri> proxies)
    {
        _proxies = proxies;
    }

    public Uri? GetNextProxy(Uri destination)
    {
        lock (_lock)
        {
            if (_proxies.Count == 0)
            {
                return null;
            }

            var proxy = _proxies[_index % _proxies.Count];
            _index++;
            return proxy;
        }
    }
}

Then configure the client once:

var provider = new RoundRobinProxyProvider(new[]
{
    new Uri("http://proxy-a.example:8080"),
    new Uri("http://proxy-b.example:8080"),
    new Uri("socks5://proxy-c.example:1080")
});

var handler = new SocketsHttpHandler
{
    UseProxy = true,
    Proxy = new RotatingProxy(provider)
};

var client = new HttpClient(handler);

After that, keep using the same client:

foreach (var url in urls)
{
    var response = await client.GetAsync(url);
    response.EnsureSuccessStatusCode();
}

Each request can now use a different proxy without recreating the HttpClient or the handler.

Failures are still normal

This does not make proxies reliable. Especially when scraping, you still get plenty of nonsense:

HttpRequestException
TaskCanceledException
connection refused
proxy tunnel failures
authentication failures
TLS failures

The difference is that these are now normal per-request failures. A broken proxy does not mean your HttpClient is broken. Catch the error, mark the proxy as bad, and try the next one.

try
{
    using var response = await client.GetAsync(url, cancellationToken);
    response.EnsureSuccessStatusCode();

    proxyHealth.ReportSuccess();
}
catch (HttpRequestException)
{
    proxyHealth.ReportFailure();
}
catch (TaskCanceledException)
{
    proxyHealth.ReportFailure();
}

In real code, you obviously want to know which proxy was used for which request, so you can update the right proxy in your database. You can solve that by having your provider hand out leases, or by keeping the selected proxy in request context.

Credentials

If all proxies use the same credentials, this is simple:

var proxy = new RotatingProxy(provider)
{
    Credentials = new NetworkCredential(username, password)
};

If proxies have different credentials, handle that a bit more carefully. Often CredentialCache is enough:

var credentials = new CredentialCache();

credentials.Add(
    new Uri("http://proxy-a.example:8080"),
    "Basic",
    new NetworkCredential("user-a", "password-a"));

credentials.Add(
    new Uri("http://proxy-b.example:8080"),
    "Basic",
    new NetworkCredential("user-b", "password-b"));

var proxy = new RotatingProxy(provider)
{
    Credentials = credentials
};

For more complex cases, you can implement ICredentials yourself.

In Dependency Injection

In DI, this becomes a lot cleaner. The proxy provider decides which proxy to use. The HTTP client stays stable.

services.AddSingleton<IProxyProvider, DatabaseBackedProxyProvider>();

services.AddHttpClient<ScraperClient>()
    .ConfigurePrimaryHttpMessageHandler(sp => new SocketsHttpHandler
    {
        UseProxy = true,
        Proxy = new RotatingProxy(sp.GetRequiredService<IProxyProvider>())
    });

Your database can change after that. Your proxy pool can grow, shrink, disable proxies, and retry them later. None of that requires registering new HTTP clients.

Tested

I tested this with one HttpClient, one SocketsHttpHandler, and a custom IWebProxy.

First with authenticated datacenter proxies. One client could use multiple ports on the same host, and the public IP changed per request.

Then with a public proxy list containing different hosts and schemes. With the same client and the same handler, requests went through different socks4:// and socks5:// proxies. Many free proxies were dead or slow, which should surprise no one. But the point was clear: the client did not need to be recreated.

Short version: if you want to switch proxies dynamically in .NET, do not change the handler and do not create a new HttpClient for every proxy. Let your IWebProxy choose the proxy per request.