Go Proxy Scraping: http.Transport, Auth, Rotation and Concurrency

In Go the whole proxy story lives on one field: Transport.Proxy. Set it right and you get per-request rotation, SOCKS5 and colly for free. Set it wrong and you get 'context canceled' at 2am.

Go makes proxying deceptively simple: the entire client-side story lives on one field, http.Transport.Proxy. Get it right and per-request rotation, SOCKS5 and colly integration all fall out of the same design. Get it wrong and you meet the classic Go proxy failures - context canceled, connection-pool exhaustion, and an HTTP_PROXY variable that your custom transport silently ignores. This guide walks the whole path with runnable code.

The one-liner: http.Transport with a proxy URL

Every proxied request in Go flows through a Transport. The net/http package ships a helper, http.ProxyURL, that pins one static proxy for the whole client. Credentials go straight into the URL userinfo - Go turns them into the Proxy-Authorization header for you:

package main

import (
  "fmt"
  "net/http"
  "net/url"
  "time"
)

func main() {
  proxyURL, _ := url.Parse("http://USER:PASS@gate.quantumproxies.io:8000")
  client := &http.Client{
    Timeout:   20 * time.Second,
    Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
  }
  resp, err := client.Get("https://httpbin.org/ip")
  if err != nil {
    panic(err)
  }
  defer resp.Body.Close()
  fmt.Println(resp.Status) // body shows the exit IP, not yours
}

Always set Client.Timeout. Go has no default timeout on http.Client, so a single dead exit will block the goroutine forever - the number-one reason a Go scraper appears to hang is a missing timeout, not a slow target.

Rotating proxies with a Proxy function

Proxy is not limited to a fixed URL - it is a function func(*http.Request) (*url.URL, error) that Go calls once per request. That is your rotation hook. Pick a random endpoint from a pool and every request leaves through a different IP, with zero list-management logic in your fetch loop:

import "math/rand"

pool := []string{
  "http://USER:PASS@ip1.quantumproxies.io:8000",
  "http://USER:PASS@ip2.quantumproxies.io:8000",
  "http://USER:PASS@ip3.quantumproxies.io:8000",
}

transport := &http.Transport{
  Proxy: func(r *http.Request) (*url.URL, error) {
    return url.Parse(pool[rand.Intn(len(pool))])
  },
  MaxIdleConnsPerHost: 32, // reuse connections across the pool
}

For most scraping you should not manage a pool at all. Point the Proxy function at a single rotating residential gateway and the gateway hands you a fresh IP on every request across 90M+ addresses in 200+ countries - one endpoint, no list, sticky sessions when a flow needs the same exit for a few minutes.

Proxy authentication in Go

Putting USER:PASS@ in the proxy URL is the clean path and works for both ProxyURL and a custom Proxy function. If you prefer to keep credentials out of the URL, set the header on the CONNECT tunnel yourself via Transport.ProxyConnectHeader. If your provider uses IP-whitelist auth instead, drop the credentials entirely and authorise your server's IP in the dashboard - QuantumProxies supports both. A refused CONNECT with 407 or proxyconnect tcp almost always means missing or wrong credentials, not a target-side block.

The HTTP_PROXY environment variable (and when Go ignores it)

Go's http.DefaultTransport uses http.ProxyFromEnvironment, which reads HTTP_PROXY, HTTPS_PROXY and NO_PROXY (per the golang.org/x/net/http/httpproxy package). Two behaviours trip people up. First, those variables accept either a full URL or a bare host:port, where the http scheme is assumed. Second, requests to localhost or a loopback address always bypass the proxy, returning a nil URL. The bigger gotcha is the reverse: the moment you build your own &http.Transport{Proxy: ...}, you have replaced the env-var behaviour - your explicit function wins and HTTP_PROXY is ignored. If you want both, wrap them yourself.

SOCKS5 proxies in Go

The standard library has no SOCKS5 client, so pull in golang.org/x/net/proxy and dial through it. Feed the dialer into Transport.DialContext so DNS resolves at the proxy, not on your machine (the same hostname-leak issue that socks5h solves in other stacks). If you are weighing the two schemes, our note on SOCKS5 vs HTTP proxies covers when each wins:

go get golang.org/x/net/proxy

import (
  "context"
  "net"
  "net/http"
  "golang.org/x/net/proxy"
)

auth := &proxy.Auth{User: "USER", Password: "PASS"}
dialer, err := proxy.SOCKS5("tcp", "gate.quantumproxies.io:1080", auth, proxy.Direct)
if err != nil {
  panic(err)
}

transport := &http.Transport{
  DialContext: func(ctx context.Context, network, addr string) (net.Conn, error) {
    return dialer.Dial(network, addr)
  },
}
client := &http.Client{Transport: transport}

Proxies in colly

If you scrape with colly, you get rotation without touching the transport. The colly/proxy package ships a RoundRobinProxySwitcher that cycles endpoints per request, and LimitRule handles pacing. This is the fastest way to a polite, rotating crawler in Go:

import (
  "time"
  "github.com/gocolly/colly/v2"
  "github.com/gocolly/colly/v2/proxy"
)

c := colly.NewCollector(colly.Async(true))

rp, err := proxy.RoundRobinProxySwitcher(
  "http://USER:PASS@ip1.quantumproxies.io:8000",
  "http://USER:PASS@ip2.quantumproxies.io:8000",
)
if err != nil {
  panic(err)
}
c.SetProxyFunc(rp)

c.Limit(&colly.LimitRule{
  DomainGlob:  "*",
  Parallelism: 8,
  RandomDelay: 2 * time.Second,
})

Concurrency done right (and the context-canceled trap)

Go's concurrency is why teams pick it for scraping, and also where the context canceled error comes from. It fires when a request's context is cancelled before the body is fully read - usually a per-request timeout that expired, or a cancel() that ran too early because of a misplaced defer. Bound your fan-out with a semaphore, give each request its own timeout context, and read then close the body before the cancel fires:

sem := make(chan struct{}, 20) // cap concurrency
var wg sync.WaitGroup

for _, u := range urls {
  wg.Add(1)
  sem <- struct{}{}
  go func(u string) {
    defer wg.Done()
    defer func() { <-sem }()

    ctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
    defer cancel()

    req, _ := http.NewRequestWithContext(ctx, "GET", u, nil)
    resp, err := client.Do(req)
    if err != nil {
      return // rotate/log; do not retry the same burned exit
    }
    io.Copy(io.Discard, resp.Body) // drain BEFORE cancel fires
    resp.Body.Close()
  }(u)
}
wg.Wait()

Two more production notes. Set MaxIdleConnsPerHost high enough that a big pool reuses connections instead of exhausting file descriptors - the usual cause of EOF after a few hundred requests. And rotate the exit on failure rather than retrying the same one; a dead residential IP does not recover on the next attempt. For the wider design at millions of pages, our guide on large-scale scraping architecture covers queues, dedup and proxy tiering.

Reuse one client, tune the transport

Create the http.Client once and share it across every goroutine - it is safe for concurrent use, and its connection pool is what makes a Go scraper fast. A fresh client per request throws away keep-alive and forces a new TLS handshake every time. Three Transport fields matter under load: MaxIdleConns and MaxIdleConnsPerHost size the pool, and IdleConnTimeout retires stale connections so a rotating proxy pool never pins a dead exit. Set them explicitly - the standard-library defaults are tuned for a browser, not a crawler running thousands of requests a minute.

When to stop hand-rolling transports

The code above is enough for clean targets. Once a site adds Cloudflare, TLS fingerprinting or JavaScript rendering, a raw Go TLS handshake looks nothing like Chrome's, and no proxy fixes that mismatch. At that point a Scraper API that carries a real browser fingerprint, rotates IPs and renders JS for you is less code and a higher success rate than maintaining the stack by hand. If you are choosing a language for a new scraper, our best proxies for web scraping post maps the trade-offs across stacks.

Offload rendering and rotation to the Scraper API

Frequently asked questions

How do I set a proxy for the Go HTTP client?

Build an http.Transport with a Proxy field and pass it to http.Client. Use http.ProxyURL(u) for a single static proxy, or a func(*http.Request) (*url.URL, error) to choose the exit per request. Put credentials in the URL as http://user:pass@host:port and always set Client.Timeout.

Why does Go ignore my HTTP_PROXY variable?

Because you built a custom Transport with an explicit Proxy function, which replaces the default ProxyFromEnvironment behaviour. Env vars only apply when you use http.DefaultTransport or set Proxy: http.ProxyFromEnvironment yourself. Also note that loopback and localhost requests always bypass the proxy.

What causes 'context canceled' with a Go proxy?

The request's context was cancelled before the response body was read - typically an expired per-request timeout or a cancel() that ran too early via defer. Give each request its own timeout context, drain and close the body before the deferred cancel executes, and don't share one cancellable context across many goroutines.

Does Go support SOCKS5 proxies?

Not in the standard library, but golang.org/x/net/proxy adds a SOCKS5 dialer. Create it with proxy.SOCKS5 and plug it into Transport.DialContext so hostnames resolve at the proxy rather than leaking through a local DNS lookup.

That is the whole client-side picture: one Transport field for setup, a function for rotation, x/net/proxy for SOCKS5, colly for polite crawling, and a semaphore plus per-request contexts for concurrency. Start with a clean rotating pool and you skip most of the error list before it happens.

Start with QuantumProxies residential proxies