# Go Proxy Scraping: http.Transport, Auth, Rotation and Concurrency

In Go the whole proxy story lives on one field: Transport.Proxy. Set it right and you get per-request rotation, SOCKS5 and colly for free. Set it wrong and you get 'context canceled' at 2am.

Go makes proxying deceptively simple: the entire client-side story lives on one field, `http.Transport.Proxy`. Get it right and per-request rotation, SOCKS5 and `colly` integration all fall out of the same design. Get it wrong and you meet the classic Go proxy failures - `context canceled`, connection-pool exhaustion, and an `HTTP_PROXY` variable that your custom transport silently ignores. This guide walks the whole path with runnable code.

## The one-liner: http.Transport with a proxy URL

Every proxied request in Go flows through a `Transport`. The `net/http` package ships a helper, `http.ProxyURL`, that pins one static proxy for the whole client. Credentials go straight into the URL userinfo - Go turns them into the `Proxy-Authorization` header for you:

```go
package main

import (
  "fmt"
  "net/http"
  "net/url"
  "time"
)

func main() {
  proxyURL, _ := url.Parse("http://USER:PASS@residentialpremium.quantumproxies.io:8080")
  client := &http.Client{
    Timeout:   20 * time.Second,
    Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
  }
  resp, err := client.Get("https://httpbin.org/ip")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()
  fmt.Println(resp.Status) // body shows the exit IP, not yours
}
```

Always set `Client.Timeout`. Go has no default timeout on `http.Client`, so a single dead exit will block the goroutine forever - the number-one reason a Go scraper appears to hang is a missing timeout, not a slow target.

## Rotating proxies with a Proxy function

`Proxy` is not limited to a fixed URL - it is a function `func(*http.Request) (*url.URL, error)` that Go calls once per request. That is your rotation hook. Pick a random endpoint from a pool and every request leaves through a different IP, with zero list-management logic in your fetch loop:

```go
import "math/rand"

// three sticky sessions on the same gateway = three exit IPs to pick from
pool := []string{
  "http://USER-sid-pool1-ttl-30m:PASS@residentialpremium.quantumproxies.io:8080",
  "http://USER-sid-pool2-ttl-30m:PASS@residentialpremium.quantumproxies.io:8080",
  "http://USER-sid-pool3-ttl-30m:PASS@residentialpremium.quantumproxies.io:8080",
}

transport := &http.Transport{
  Proxy: func(r *http.Request) (*url.URL, error) {
    return url.Parse(pool[rand.Intn(len(pool))])
  },
  MaxIdleConnsPerHost: 32, // reuse connections across the pool
}
```

For most scraping you should not manage a pool at all. Point the `Proxy` function at a single [rotating residential gateway](https://quantumproxies.io/rotating-proxies) and the gateway hands you a fresh IP on every request across 90M+ addresses in 200+ countries - one endpoint, no list, sticky sessions when a flow needs the same exit for a few minutes.

![One Go http.Client with one Transport whose Proxy function runs for every request, so requests 1, 2 and 3 leave through different exit IPs to the same target. The card: the function is called once per request, picks from a pool or points at one rotating gateway, and MaxIdleConnsPerHost: 32 keeps connections reused.](https://res.cloudinary.com/dmqliab0t/image/upload/f_auto,q_auto,w_1200/v1785335148/blog/generated/qp-b9-009-diag1.png)
*One Transport, one Proxy function: the function runs per request, so a single client rotates across the whole pool.*

## Proxy authentication in Go

Putting `USER:PASS@` in the proxy URL is the clean path and works for both `ProxyURL` and a custom `Proxy` function. If you prefer to keep credentials out of the URL, set the header on the CONNECT tunnel yourself via `Transport.ProxyConnectHeader`. If your provider uses IP-whitelist auth instead, drop the credentials entirely and authorise your server's IP in the dashboard - QuantumProxies supports both. A refused CONNECT with `407` or `proxyconnect tcp` almost always means missing or wrong credentials, not a target-side block.

## The HTTP_PROXY environment variable (and when Go ignores it)

Go's `http.DefaultTransport` uses `http.ProxyFromEnvironment`, which reads `HTTP_PROXY`, `HTTPS_PROXY` and `NO_PROXY` (per the `golang.org/x/net/http/httpproxy` package). Two behaviours trip people up. First, those variables accept either a full URL or a bare `host:port`, where the `http` scheme is assumed. Second, requests to `localhost` or a loopback address always bypass the proxy, returning a nil URL. The bigger gotcha is the reverse: the moment you build your own `&http.Transport{Proxy: ...}`, you have *replaced* the env-var behaviour - your explicit function wins and `HTTP_PROXY` is ignored. If you want both, wrap them yourself.

## SOCKS5 proxies in Go

The standard library has no SOCKS5 client, so pull in `golang.org/x/net/proxy` and dial through it. Feed the dialer into `Transport.DialContext` so DNS resolves at the proxy, not on your machine (the same hostname-leak issue that `socks5h` solves in other stacks). If you are weighing the two schemes, our note on [SOCKS5 vs HTTP proxies](https://quantumproxies.io/blog/socks5-vs-http-proxy-difference) covers when each wins:

```bash
go get golang.org/x/net/proxy
```

```go
import (
  "context"
  "net"
  "net/http"
  "golang.org/x/net/proxy"
)

auth := &proxy.Auth{User: "USER", Password: "PASS"}
dialer, err := proxy.SOCKS5("tcp", "residentialpremium.quantumproxies.io:1080", auth, proxy.Direct)
if err != nil {
  panic(err)
}

transport := &http.Transport{
  DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
    return dialer.Dial(network, addr)
  },
}
client := &http.Client{Transport: transport}
```

## Proxies in colly

If you scrape with `colly`, you get rotation without touching the transport. The `colly/proxy` package ships a `RoundRobinProxySwitcher` that cycles endpoints per request, and `LimitRule` handles pacing. This is the fastest way to a polite, rotating crawler in Go:

```go
import (
  "time"
  "github.com/gocolly/colly/v2"
  "github.com/gocolly/colly/v2/proxy"
)

c := colly.NewCollector(colly.Async(true))

rp, err := proxy.RoundRobinProxySwitcher(
  "http://USER-sid-crawl1-ttl-30m:PASS@residentialpremium.quantumproxies.io:8080",
  "http://USER-sid-crawl2-ttl-30m:PASS@residentialpremium.quantumproxies.io:8080",
)
if err != nil {
  panic(err)
}
c.SetProxyFunc(rp)

c.Limit(&colly.LimitRule{
  DomainGlob:  "*",
  Parallelism: 8,
  RandomDelay: 2 * time.Second,
})
```

## Concurrency done right (and the context-canceled trap)

Go's concurrency is why teams pick it for scraping, and also where the `context canceled` error comes from. It fires when a request's context is cancelled before the body is fully read - usually a per-request timeout that expired, or a `cancel()` that ran too early because of a misplaced `defer`. Bound your fan-out with a semaphore, give each request its own timeout context, and read then close the body before the cancel fires:

```go
sem := make(chan struct{}, 20) // cap concurrency
var wg sync.WaitGroup

for _, u := range urls {
  wg.Add(1)
  sem <- struct{}{}
  go func(u string) {
    defer wg.Done()
    defer func() { <-sem }()

    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    req, _ := http.NewRequestWithContext(ctx, "GET", u, nil)
    resp, err := client.Do(req)
    if err != nil {
      return // rotate/log; do not retry the same burned exit
    }
    io.Copy(io.Discard, resp.Body) // drain BEFORE cancel fires
    resp.Body.Close()
  }(u)
}
wg.Wait()
```

Two more production notes. Set `MaxIdleConnsPerHost` high enough that a big pool reuses connections instead of exhausting file descriptors - the usual cause of `EOF` after a few hundred requests. And rotate the exit on failure rather than retrying the same one; a dead residential IP does not recover on the next attempt. For the wider design at millions of pages, our guide on [large-scale scraping architecture](https://quantumproxies.io/blog/large-scale-web-scraping-architecture) covers queues, dedup and proxy tiering.

## Reuse one client, tune the transport

Create the `http.Client` once and share it across every goroutine - it is safe for concurrent use, and its connection pool is what makes a Go scraper fast. A fresh client per request throws away keep-alive and forces a new TLS handshake every time. Three `Transport` fields matter under load: `MaxIdleConns` and `MaxIdleConnsPerHost` size the pool, and `IdleConnTimeout` retires stale connections so a rotating proxy pool never pins a dead exit. Set them explicitly - the standard-library defaults are tuned for a browser, not a crawler running thousands of requests a minute.

![A table of Go proxy errors with the cause column highlighted. The card maps each error to its cause: proxyconnect or 407 means bad credentials, context canceled means cancel() ran before the body was read, EOF means MaxIdleConnsPerHost is too low, an ignored HTTP_PROXY means a custom Transport; a hang, no timeout.](https://res.cloudinary.com/dmqliab0t/image/upload/f_auto,q_auto,w_1200/v1785335148/blog/generated/qp-b9-009-diag2.png)
*Nearly every Go proxy failure maps to one of these five - check the cause column before you retry the same exit.*

## When to stop hand-rolling transports

The code above is enough for clean targets. Once a site adds Cloudflare, TLS fingerprinting or JavaScript rendering, a raw Go TLS handshake looks nothing like Chrome's, and no proxy fixes that mismatch. At that point a [Scraper API](https://quantumproxies.io/scraper-api) that carries a real browser fingerprint, rotates IPs and renders JS for you is less code and a higher success rate than maintaining the stack by hand. If you are choosing a language for a new scraper, our [best proxies for web scraping](https://quantumproxies.io/blog/best-proxies-for-web-scraping) post maps the trade-offs across stacks.

[Offload rendering and rotation to the Scraper API](https://quantumproxies.io/scraper-api)

## Frequently asked questions

### How do I set a proxy for the Go HTTP client?

Build an `http.Transport` with a `Proxy` field and pass it to `http.Client`. Use `http.ProxyURL(u)` for a single static proxy, or a `func(*http.Request) (*url.URL, error)` to choose the exit per request. Put credentials in the URL as `http://user:pass@host:port` and always set `Client.Timeout`.

### Why does Go ignore my HTTP_PROXY variable?

Because you built a custom `Transport` with an explicit `Proxy` function, which replaces the default `ProxyFromEnvironment` behaviour. Env vars only apply when you use `http.DefaultTransport` or set `Proxy: http.ProxyFromEnvironment` yourself. Also note that loopback and localhost requests always bypass the proxy.

### What causes 'context canceled' with a Go proxy?

The request's context was cancelled before the response body was read - typically an expired per-request timeout or a `cancel()` that ran too early via `defer`. Give each request its own timeout context, drain and close the body before the deferred cancel executes, and don't share one cancellable context across many goroutines.

### Does Go support SOCKS5 proxies?

Not in the standard library, but `golang.org/x/net/proxy` adds a SOCKS5 dialer. Create it with `proxy.SOCKS5` and plug it into `Transport.DialContext` so hostnames resolve at the proxy rather than leaking through a local DNS lookup.

That is the whole client-side picture: one `Transport` field for setup, a function for rotation, `x/net/proxy` for SOCKS5, `colly` for polite crawling, and a semaphore plus per-request contexts for concurrency. Start with a clean rotating pool and you skip most of the error list before it happens.

[Start with QuantumProxies residential proxies](https://quantumproxies.io/residential-proxies)