Go Microservices at 10M Concurrent Connections

Most "Go vs. epoll" debates are framed badly. They pose goroutines and event loops as rival concurrency models, when in reality Go's netpoller is an epoll loop — just one that the runtime hides behind blocking-looking APIs. The honest question is not which mechanism you use, but who owns the per-connection state machine and how much memory that ownership costs when you have ten million of them.

This article dissects that question with production numbers: how a Go microservice behaves at 1M, 5M, and 10M concurrent sockets, where the memory actually goes (user space and kernel), why p99 degrades non-linearly, and how to compute cost per million connections so you can make a defensible capacity decision instead of a vibe-based one.

The Real Question: Who Owns the State Machine?

There are only two things you can do with 10M sockets:

1. Give each one a goroutine. The Go scheduler multiplexes ~10M goroutines onto GOMAXPROCS OS threads. Each goroutine is a linear piece of code that blocks on Read(). Cost: a stack, a g struct, plus whatever your handler allocates.
2. Give each one a callback struct. A fixed number of event-loop threads call epoll_wait, and each socket is a small heap object with explicit state. Cost: your own buffers and your own state machine.

Go's runtime does the first, and it does the second underneath it — the netpoller is the bridge. This is why the model is not actually "goroutine-per-request" at connection scale: for HTTP/1.1 keep-alive, it is goroutine-per-connection, parked on epoll_wait inside runtime.netpoll and resumed by goready() when a file descriptor fires.

How the netpoller wakes a goroutine

When your handler blocks on a socket read:

• net.(*conn).Read → internal/poll.(*FD).Read → syscall.Read returns EAGAIN.
• poll.FD.Read calls runtime.netpollblock, which sets the goroutine's state to _Gwaiting and calls gopark with the fd recorded in the pollDesc.
• The fd is registered with epoll_ctl(EPOLL_CTL_ADD, EPOLLIN|EPOLLONESHOT|EPOLLRDHUP).
• When data arrives, epoll_wait in the netpoller returns the fd, netpollready calls netpollunblock, the goroutine moves to _Grunnable and is pushed onto a P's local run queue (or its runnext slot for lower latency).

That round-trip — readiness → runnable → scheduled → executing — is the entire p99 story for Go at scale. There is no syscall wakeup per connection, no thread-per-connection, and no blocking accept loop. Which means the only thing that distinguishes a Go server's tail latency from a hand-written epoll server's is (a) scheduler run-queue depth, (b) GC assist work, and (c) memory footprint.

The Anatomy of a Go Connection at 10M Scale

User-space: what net/http actually allocates per idle keep-alive socket

This is the table that decides your instance size. All figures are per idle HTTP/1.1 keep-alive connection on linux/amd64.

At 10M connections, that's ~110 GiB vs. ~24 GiB of user-space RSS before the kernel's contribution. This is the single most important number in the article, and it is why a 256 GiB memory-optimized instance can hold 10M Go connections but not 20M.

The bufio.Reader/bufio.Writer pairs are non-negotiable in net/http — the defaults are 4 KiB each and cannot go below 4 KiB without hurting throughput on real request bodies. If you shrink ReadBufferSize to 1 KiB you'll trade 6 KiB of heap for a meaningful throughput regression on anything larger than a header. A hand-written epoll loop lets you size ring buffers to your actual protocol (often 512 B–1 KiB for connection-oriented control planes).

Kernel-space: tcp_sock, buffers, and the tcp_mem ceiling

User-space RSS is only half the bill. Every socket also costs:

• struct tcp_sock (~2.3 KiB, embedded in inet_connection_sock) + struct socket + struct file + slab overhead ≈ 2.7 KiB
• Receive/send buffers. net.ipv4.tcp_rmem defaults to 4096 131072 6291456 on most distros. Autotuning collapses idle buffers toward the minimum, but plan 3–8 KiB per idle socket in the kernel's TCP memory accounting.

That second bullet is the trap. Linux caps total TCP memory via net.ipv4.tcp_mem, whose defaults are computed at boot as roughly 4.7% / 6.25% / 9.4% of available RAM. On a 256 GiB host the hard ceiling is around 24 GiB. Ten million sockets at 4 KiB each demand 40 GiB — you will hit Out of socket memory in dmesg and stare at SYN drops long before you hit CPU limits.

# Required kernel-side headroom for 10M sockets on a 256 GiB host
sysctl -w net.ipv4.tcp_mem='26214400 31457280 39321600'   # pages (~100/120/150 GiB)
sysctl -w net.ipv4.tcp_rmem='4096 16384 4194304'           # shrink idle rcvbuf
sysctl -w net.ipv4.tcp_wmem='4096 16384 4194304'
sysctl -w net.core.rmem_max=16777216
sysctl -w net.core.somaxconn=65535
sysctl -w net.ipv4.tcp_max_syn_backlog=65535
sysctl -w net.core.netdev_max_backlog=250000
sysctl -w fs.nr_open=25000000                              # per-process NOFILE cap
sysctl -w fs.file-max=50000000

And in systemd: LimitNOFILE=20000000. Note that fs.nr_open is the hard ceiling for RLIMIT_NOFILE; raising LimitNOFILE above it silently clamps.

Tuning the Listener: Production-Grade Socket Options

The difference between a listener that survives a 10M-connection storm and one that melts is a handful of setsockopt calls. Here's a complete, drop-in net.ListenConfig with proper error propagation.

package transport

import (
	"context"
	"errors"
	"fmt"
	"net"
	"syscall"
	"time"

	"golang.org/x/sys/unix"
)

// ListenC10M opens a TCP listener tuned for very high concurrent socket counts.
// Every socket option is best-effort-joined so a partial failure is still reported.
func ListenC10M(ctx context.Context, addr string) (net.Listener, error) {
	lc := net.ListenConfig{
		Control: func(network, address string, c syscall.RawConn) error {
			var optErr error
			ctlErr := c.Control(func(fd uintptr) {
				f := int(fd)
				// SO_REUSEPORT: N accept loops (or N pods) share one port; the
				// kernel hashes flows across them, doubling throughput per core.
				if err := unix.SetsockoptInt(f, unix.SOL_SOCKET, unix.SO_REUSEPORT, 1); err != nil {
					optErr = errors.Join(optErr, fmt.Errorf("SO_REUSEPORT: %w", err))
				}
				// SO_REUSEADDR: bind during TIME_WAIT churn on rolling restarts.
				if err := unix.SetsockoptInt(f, unix.SOL_SOCKET, unix.SO_REUSEADDR, 1); err != nil {
					optErr = errors.Join(optErr, fmt.Errorf("SO_REUSEADDR: %w", err))
				}
				// TCP_DEFER_ACCEPT: don't wake a goroutine until the peer sends data.
				// Critical for idle-connection fleets that connect eagerly.
				if err := unix.SetsockoptInt(f, unix.IPPROTO_TCP, unix.TCP_DEFER_ACCEPT, 1); err != nil {
					optErr = errors.Join(optErr, fmt.Errorf("TCP_DEFER_ACCEPT: %w", err))
				}
				// TCP_FASTOPEN: remove a full RTT from reconnect storms.
				if err := unix.SetsockoptInt(f, unix.IPPROTO_TCP, unix.TCP_FASTOPEN, 1024); err != nil {
					optErr = errors.Join(optErr, fmt.Errorf("TCP_FASTOPEN: %w", err))
				}
			})
			if ctlErr != nil {
				return fmt.Errorf("raw conn control: %w", ctlErr)
			}
			return optErr
		},
		KeepAlive: 30 * time.Second,
		KeepAliveConfig: net.KeepAliveConfig{
			Enable:   true,
			Idle:     60 * time.Second,
			Interval: 15 * time.Second,
			Count:    4,
		},
	}

	ln, err := lc.Listen(ctx, "tcp", addr)
	if err != nil {
		return nil, fmt.Errorf("listen %s: %w", addr, err)
	}
	return ln, nil
}

Pair this with an http.Server that has explicit timeouts (ReadHeaderTimeout, IdleTimeout, MaxHeaderBytes) — an unset ReadHeaderTimeout is a slowloris invitation, and an unset IdleTimeout means your 10M connections are 10M permanent memory residents.

Building the Alternative: A Raw epoll Event Loop

If you need the 8 KiB/connection back, you have to write the state machine yourself. Here's the skeleton of a production-shaped epoll loop — the same architecture gnet and Netty use.

//go:build linux

package evloop

import (
	"errors"
	"fmt"
	"sync/atomic"

	"golang.org/x/sys/unix"
)

// Handler is the per-connection state machine. It is called on the loop
// goroutine and MUST NOT block.
type Handler interface {
	OnReadable(c *Conn) error
	OnWritable(c *Conn) error
	OnClose(c *Conn)
}

type Conn struct {
	fd   int
	loop *Loop
	h    Handler
}

type Loop struct {
	epfd    int
	evfd    int
	conns   map[int]*Conn
	events  []unix.EpollEvent
	running atomic.Bool
}

func NewLoop() (*Loop, error) {
	epfd, err := unix.EpollCreate1(unix.EPOLL_CLOEXEC)
	if err != nil {
		return nil, fmt.Errorf("epoll_create1: %w", err)
	}
	evfd, err := unix.Eventfd(0, unix.EFD_NONBLOCK|unix.EFD_CLOEXEC)
	if err != nil {
		unix.Close(epfd)
		return nil, fmt.Errorf("eventfd: %w", err)
	}
	if err := unix.EpollCtl(epfd, unix.EPOLL_CTL_ADD, evfd,
		&unix.EpollEvent{Events: unix.EPOLLIN, Fd: int32(evfd)}); err != nil {
		unix.Close(evfd)
		unix.Close(epfd)
		return nil, fmt.Errorf("epoll_ctl(evfd): %w", err)
	}
	return &Loop{
		epfd:   epfd,
		evfd:   evfd,
		conns:  make(map[int]*Conn, 1<<20), // pre-size for millions of entries
		events: make([]unix.EpollEvent, 4096),
	}, nil
}

// accept drains the listener in edge-triggered style; it only returns on
// EAGAIN (normal), fd exhaustion (fatal), or an unrecoverable error.
func (l *Loop) accept(lfd int, mk func(fd int) Handler) error {
	for {
		nfd, _, err := unix.Accept4(lfd, unix.SOCK_NONBLOCK|unix.SOCK_CLOEXEC)
		if err != nil {
			switch {
			case errors.Is(err, unix.EAGAIN), errors.Is(err, unix.EWOULDBLOCK):
				return nil
			case errors.Is(err, unix.EINTR), errors.Is(err, unix.ECONNABORTED):
				continue
			case errors.Is(err, unix.EMFILE), errors.Is(err, unix.ENFILE):
				return fmt.Errorf("accept4: fd table exhausted: %w", err)
			default:
				return fmt.Errorf("accept4: %w", err)
			}
		}

		c := &Conn{fd: nfd, loop: l, h: mk(nfd)}
		l.conns[nfd] = c
		if err := unix.EpollCtl(l.epfd, unix.EPOLL_CTL_ADD, nfd, &unix.EpollEvent{
			Events: unix.EPOLLIN | unix.EPOLLET | unix.EPOLLRDHUP,
			Fd:     int32(nfd),
		}); err != nil {
			delete(l.conns, nfd)
			unix.Close(nfd)
			return fmt.Errorf("epoll_ctl(add %d): %w", nfd, err)
		}
	}
}

func (l *Loop) close(c *Conn) {
	c.h.OnClose(c)
	delete(l.conns, c.fd)
	_ = unix.Close(c.fd)
}

// Run is the single-threaded event loop. Pin one Loop per OS thread.
func (l *Loop) Run() error {
	l.running.Store(true)
	for l.running.Load() {
		n, err := unix.EpollWait(l.epfd, l.events, -1)
		if err != nil {
			if errors.Is(err, unix.EINTR) {
				continue
			}
			return fmt.Errorf("epoll_wait: %w", err)
		}
		for i := 0; i < n; i++ {
			ev := &l.events[i]
			if int(ev.Fd) == l.evfd {
				var buf [8]byte
				_, _ = unix.Read(l.evfd, buf[:]) // drain shutdown signal
				continue
			}
			c, ok := l.conns[int(ev.Fd)]
			if !ok {
				continue
			}
			if ev.Events&(unix.EPOLLERR|unix.EPOLLHUP) != 0 {
				l.close(c)
				continue
			}
			if ev.Events&unix.EPOLLIN != 0 {
				if err := c.h.OnReadable(c); err != nil {
					l.close(c)
					continue
				}
			}
			if ev.Events&unix.EPOLLOUT != 0 {
				if err := c.h.OnWritable(c); err != nil {
					l.close(c)
				}
			}
		}
	}
	return nil
}

Note the EPOLLET (edge-triggered) flag: your OnReadable must loop on read() until EAGAIN, or you will silently stall connections. This is the single most common bug in home-grown epoll servers, and it is exactly the class of bug the goroutine model eliminates by construction.

Benchmark Methodology That Doesn't Lie

Before any numbers, a methodology warning: most c10m benchmarks are wrong because the load generator becomes the bottleneck. To generate 10M concurrent TCP connections you need:

• ≥ 8 load-generator hosts, because a single source IP gives you only ~64K ephemeral ports per destination 4-tuple. Ten million connections require ~150 source IPs minimum; practically, use 8–16 generator machines each with multiple IPs.
• `SO_REUSEPORT` on the server listener so the kernel distributes accepts across all GOMAXPROCS without a single accept lock hotspot.
• Zero-loss measurement. Read /proc/net/softnet_stat column 2 (dropped) on every host; if it's non-zero, your p99 is measuring the NIC, not the application.
• Measure RSS from `/proc/<pid>/status` `VmRSS`, not Go's heap metrics — runtime.MemStats.HeapInuse misses goroutine stacks, span overhead, and the entire kernel side.

Illustrative results (64 vCPU Graviton3, 256 GiB, 8 generator hosts)

Read these as directional, not absolute. The pattern matters more than the digits:

• At 1M connections, the two models are within noise on p99 — goroutines are genuinely free.
• The goroutine curve bends sharply after ~5M because the scheduler's run-queue latency grows with the number of concurrently runnable goroutines, and GC assist work scales with a ~110 GiB live heap.
• The epoll curve stays near-linear because its live heap is ~24 GiB, keeping GC off the critical path, and each loop's run-queue is bounded by its own shard.

The Cost Model: Dollars per Million Connections

An m7g.16xlarge (64 vCPU, 256 GiB) lists at roughly $2.59/hr on-demand in us-east-1. Normalize:

Add a 30% Savings Plan discount and the epoll model lands near $53/month per million connections vs. $132/month for net/http. At 5M connections that's a ~$470/month delta per service — real, but far less than the engineering cost of maintaining a hand-rolled protocol state machine.

The decisive cost is usually not compute. It's operational: an epoll server with a subtle edge-trigger bug causes a 3 a.m. incident that costs more than a year of instance savings.

Where Goroutine-per-Connection Wins Decisively

Do not read the table above as "always write an event loop." Goroutines win outright when:

• You need `net/http`, gRPC, or `database/sql`. Every mainstream Go library is written for blocking semantics. Wrapping them in an event loop means runtime.LockOSThread gymnastics and deadlock-prone callback inversion.
• Connection count is under ~2M. At 11 KiB/conn that's ~22 GiB — well within a single node, and the scheduler is nowhere near saturation.
• Your handlers are CPU-trivial but I/O-frequent. The netpoller's wakeup cost (~1–2 µs) is irrelevant next to a downstream RPC.
• You value debuggability. A goroutine dump of 10M stacks is useless, but pprof on a sharded fleet of 200K-goroutine pods is not.

Operational Checklist Before You Chase c10m

• fs.nr_open and LimitNOFILE both raised above your target connection count, verified with cat /proc/$(pgrep myservice)/limits.
• net.ipv4.tcp_mem raised so kernel TCP accounting cannot OOM before your heap does.
• net.ipv4.tcp_rmem/tcp_wmem minima reduced to shrink idle socket buffers.
• SO_REUSEPORT enabled so accepts scale across cores and pods.
• nf_conntrack_max raised (default 262144!) or conntrack bypassed via NOTRACK rules — this silently kills c10m fleets behind NAT.
• GOMAXPROCS explicitly set (or automaxprocs/Go 1.25 container-aware defaults confirmed) in Kubernetes.
• GOMEMLIMIT set below the container limit to prevent GC-driven OOMKills.
• http.Server.ReadHeaderTimeout and IdleTimeout set — unbounded idle connections are unbounded memory.
• Load generator has ≥ 8 hosts and ≥ 150 source IPs; softnet_stat drops verified at zero.
• p99 measured at /proc/<pid>/status RSS steady state, not at connection ramp.
• RPS/RFS configured, or ksoftirqd pins a core at 100% and fakes a latency regression.

Pitfalls That Will Bite You

1. The `tcp_mem` ceiling. Your Go heap is fine; the kernel refuses the connection. Symptom: Out of socket memory in dmesg and SYN retransmits you'll misattribute to the network.

2. Conntrack exhaustion. nf_conntrack: table full, dropping packet — a classic at 262144 default entries. Every dropped packet becomes a reconnect, which becomes an accept storm, which becomes a thundering herd.

3. Edge-triggered partial reads. With EPOLLET, a single read() per event loses data under load only sometimes — the worst kind of bug.

4. GC assist spikes at large heaps. At ~110 GiB live, GOGC=100 means a 110 GiB mark cycle. Goroutines performing assist work have inflated, unbounded tail latency. GOMEMLIMIT plus a lower GOGC gives you a smooth curve instead of cliffs.

5. Assuming `GOMAXPROCS` is correct in Kubernetes. Before Go 1.25, GOMAXPROCS read host CPU count, not the cgroup quota — a 4-CPU-limited pod on a 64-core node spawns 64 Ps and thrashes. Set it explicitly.

6. Measuring heap instead of RSS. HeapInuse excludes goroutine stacks. You will size your fleet 30% too small.

7. Ignoring `SO_REUSEPORT` skew. The kernel hashes 4-tuples; with few long-lived connections per loop, distribution can be badly skewed. Monitor per-loop accept counts.

Sources & Technical References

• Go runtime source — netpoller implementation: `runtime/netpoll.go` and `runtime/netpoll_epoll.go` (golang/go, BSD license).
• Go 1.4 release notes — goroutine stacks reduced from 8 KiB to 2 KiB: go.dev/doc/go1.4#runtime.
• Go proposal #24543 — Non-cooperative goroutine preemption (async preemption, Go 1.14): github.com/golang/proposal/blob/master/design/24543-non-cooperative-preemption.md.
• `net.ListenConfig` and `net.KeepAliveConfig` official docs: pkg.go.dev/net#ListenConfig.
• `http.Server` buffer and timeout semantics: pkg.go.dev/net/http#Server.
• `epoll(7)` — Linux Programmer's Manual (edge-triggered semantics, EPOLLET, EPOLLRDHUP): man7.org/linux/man-pages/man7/epoll.7.html.
• Linux kernel IP sysctl reference (tcp_mem, tcp_rmem, tcp_wmem, somaxconn): kernel.org/doc/Documentation/networking/ip-sysctl.txt.
• RFC 9293 — Transmission Control Protocol (TCP): datatracker.ietf.org/doc/html/rfc9293.
• Dan Kegel, "The C10K Problem" (the original scaling taxonomy): kegel.com/c10k.html.
• Robert Graham, "The C10M Problem" manifesto: c10m.robertgraham.com/p/manifesto.html.
• Cloudflare Engineering, "How to receive a million packets per second" (RPS/RFS, SO_REUSEPORT, softirq behaviour): blog.cloudflare.com/how-to-receive-a-million-packets.
• `gnet` — a production Go event-loop networking library (reference implementation of the pattern above): github.com/panjf2000/gnet.
• Netty — epoll transport design notes (canonical event-loop architecture): netty.io/wiki/native-transports.html.
• `golang.org/x/sys/unix` package documentation: pkg.go.dev/golang.org/x/sys/unix.