Go at 100K RPS: Concurrency Patterns, Channels, and Memory Optimization for High-Throughput Microservices

Why Goroutines Are Cheap — and Why "Cheap" Is a Trap

A goroutine begins life with a 2 KiB stack that grows and shrinks by copying, versus the 1–8 MiB virtual reservation of a POSIX thread. The Go runtime's GMP scheduler multiplexes goroutines (G) onto OS threads (M) through logical processors (P), each holding a run queue drained by work-stealing (Blumofe & Leiserson, JACM 1999). When a goroutine enters a blocking syscall, the runtime hands the P to another M so the core keeps executing Go code. Creation costs roughly 200–300 ns; a goroutine switch is tens of nanoseconds.

That economics is precisely the hazard. go fn() is free until it isn't. Consider a service at 50K RPS that fans out to three downstream dependencies per request, with a dependency that degrades from 5 ms to 500 ms. In-flight goroutines converge on RPS × latency × fanout — roughly 75,000 concurrent goroutines. Each carries a grown stack (~8 KiB after a few frames), a decoded request, gRPC client-side buffers, and a context chain: call it 12 KiB. That is ~900 MiB of live heap plus a GC that must scan every pointer in it. The service doesn't crash from CPU; it dies from GC assist and OOMKill.

The lesson: concurrency in Go must be bounded by construction, not by discipline.

Backpressure by Construction: Bounded Worker Pools

A correct pool bounds two independent quantities: in-flight work (worker count) and queued work (channel capacity). Submit blocking is the backpressure mechanism — it propagates through the gRPC handler into HTTP/2 flow control, which pushes back on the client.

package worker

import (
	"context"
	"errors"
	"sync"
)

var ErrPoolClosed = errors.New("worker: pool closed")

// Job is a unit of work. Run must honour ctx; the pool never interrupts it.
type Job struct {
	ID  string
	Run func(ctx context.Context) error
}

// Pool bounds in-flight work (workers) and queued work (queue). Submit blocking
// is the backpressure mechanism — never give it an unbounded queue.
type Pool struct {
	jobs  chan Job
	quit  chan struct{}
	ctx   context.Context
	onErr func(error)
	wg    sync.WaitGroup
	once  sync.Once
}

func NewPool(ctx context.Context, workers, queue int, onErr func(error)) *Pool {
	p := &Pool{
		jobs:  make(chan Job, queue),
		quit:  make(chan struct{}),
		ctx:   ctx,
		onErr: onErr,
	}
	p.wg.Add(workers)
	for i := 0; i < workers; i++ {
		go p.loop()
	}
	return p
}

func (p *Pool) loop() {
	defer p.wg.Done()
	for {
		select {
		case <-p.quit:
			return
		case j := <-p.jobs:
			if err := j.Run(p.ctx); err != nil {
				p.onErr(err) // logger + metric, never a panic
			}
		}
	}
}

// Submit blocks until the job is accepted, the pool stops, or ctx is done.
func (p *Pool) Submit(ctx context.Context, j Job) error {
	select {
	case p.jobs <- j:
		return nil
	case <-p.quit:
		return ErrPoolClosed
	case <-ctx.Done():
		return ctx.Err()
	}
}

// Shutdown stops the workers. Queued jobs are abandoned by design: callers still
// observe cancellation through their own contexts.
func (p *Pool) Shutdown(ctx context.Context) error {
	p.once.Do(func() { close(p.quit) })
	done := make(chan struct{})
	go func() { p.wg.Wait(); close(done) }()
	select {
	case <-done:
		return nil
	case <-ctx.Done():
		return ctx.Err()
	}
}

Note the deliberate design decision: `jobs` is never closed. Closing a channel with concurrent senders is a guaranteed panic, and a sync.RWMutex around Submit introduces a deadlock when the queue is full and workers are stuck. Signalling shutdown through a separate quit channel keeps Submit lock-free and panic-free.

Fan-Out/Fan-In: Aggregating gRPC Calls Without Melting Downstreams

For per-request fan-out (as opposed to a long-lived pool), golang.org/x/sync/errgroup with SetLimit is the idiomatic tool. It gives you bounded parallelism, first-error cancellation, and a single Wait.

func (s *Aggregator) Aggregate(ctx context.Context, req *pb.AggregateRequest) (*pb.AggregateResponse, error) {
	ctx, cancel := context.WithTimeout(ctx, 250*time.Millisecond)
	defer cancel()

	var (
		mu   sync.Mutex
		resp = &pb.AggregateResponse{Items: make([]*pb.Item, 0, 64)} // preallocate
	)
	g, ctx := errgroup.WithContext(ctx)
	g.SetLimit(s.maxParallel) // caps goroutines; excess g.Go calls block

	for _, shard := range req.GetShards() {
		shard := shard // redundant on Go 1.22+, required below
		g.Go(func() error {
			r, err := s.inventory.Get(ctx, &pb.GetRequest{Shard: shard})
			if err != nil {
				return fmt.Errorf("inventory shard %s: %w", shard, err)
			}
			mu.Lock()
			resp.Items = append(resp.Items, r.GetItems()...)
			mu.Unlock()
			return nil
		})
	}

	if err := g.Wait(); err != nil {
		if errors.Is(err, context.DeadlineExceeded) {
			return nil, status.Error(codes.DeadlineExceeded, "aggregate deadline exceeded")
		}
		st, ok := status.FromError(err)
		if !ok {
			return nil, status.Error(codes.Internal, "aggregate failed")
		}
		return nil, st.Err()
	}
	return resp, nil
}

Three production notes. First, errgroup.WithContext cancels sibling calls on the first error — that is the correct semantic for an aggregation, but it means one shard's NotFound aborts the whole response; use g.Go returning nil for idempotent partial results if you want best-effort. Second, status.FromError unwraps %w-wrapped gRPC errors only in recent grpc-go releases — verify against your pinned version, or return st.Err() unwrapped from the innermost call. Third, a 250 ms deadline is a contract: it must be shorter than the caller's deadline, which must be shorter than the HTTP edge timeout. Deadline budgets allocate top-down, never bottom-up.

Context Lifecycle: The Goroutine and Timer Leaks Nobody Profiles

// Leaky: the timer survives until the deadline even if the RPC returned early.
for _, shard := range shards {
	ctx, _ := context.WithTimeout(parent, 200*time.Millisecond)
	call(ctx, shard)
}

// Correct: release the timer the moment the call returns.
for _, shard := range shards {
	ctx, cancel := context.WithTimeout(parent, 200*time.Millisecond)
	err := call(ctx, shard)
	cancel()
	_ = err
}

go vet's lostcancel analyser catches the obvious cases, and it belongs in CI. Two harder rules: never store a context.Context in a struct (pass it as the first parameter, per the Go blog post Contexts and Structs), and never spawn go func(){...}() without a select on ctx.Done() — a goroutine blocked on a channel send that nobody reads is a permanent leak, invisible to the heap profile until it isn't.

Channel Semantics That Bite in Production

A production-grade merge that disables drained inputs via nil channels and never leaks on cancellation:

func merge(ctx context.Context, a, b <-chan int) <-chan int {
	out := make(chan int)
	go func() {
		defer close(out)
		for a != nil || b != nil {
			select {
			case v, ok := <-a:
				if !ok {
					a = nil // disable this case; nil channels block forever
					continue
				}
				select {
				case out <- v:
				case <-ctx.Done():
					return
				}
			case v, ok := <-b:
				if !ok {
					b = nil
					continue
				}
				select {
				case out <- v:
				case <-ctx.Done():
					return
				}
			}
		}
	}()
	return out
}

Memory Optimization: Escape Analysis, Pooling, and GC Pressure

Start with evidence, not instinct:

go build -gcflags='-m=2' ./... 2>&1 | grep -E 'escapes to heap|moved to heap'
go test -bench=. -benchmem -memprofile=mem.out ./...
go tool pprof -sample_index=alloc_objects -top mem.out

alloc_objects finds hot allocation sites; alloc_space finds hot bytes; inuse_space finds what actually survives GC. Optimizing the wrong profile is how teams spend a sprint shaving 3% off CPU.

Three levers matter at scale. Preallocate: make([]Item, 0, n) eliminates repeated doubling copies. Pool: sync.Pool absorbs bursty transient allocations — but it is cleared every GC cycle, so it is not a cache. Reduce pointer density: GC scan cost is proportional to the number of pointers, so []Item is dramatically cheaper than []*Item for large collections.

const maxPooledBuffer = 1 << 20 // 1 MiB ceiling; don't cache pathological buffers

var bufPool = sync.Pool{
	New: func() any { return &proto.Buffer{} },
}

// WriteFrame serializes v into a pooled buffer and writes it before returning it
// to the pool. Never hand the pooled slice back to a caller — it will be reused.
func WriteFrame(w io.Writer, v proto.Message) error {
	buf := bufPool.Get().(*proto.Buffer)
	defer func() {
		buf.Reset()
		if cap(buf.Bytes()) <= maxPooledBuffer {
			bufPool.Put(buf)
		}
	}()
	if err := buf.Marshal(v); err != nil {
		return fmt.Errorf("marshal frame: %w", err)
	}
	if _, err := w.Write(buf.Bytes()); err != nil {
		return fmt.Errorf("write frame: %w", err)
	}
	return nil
}

Finally, tune the runtime for containers. GOMAXPROCS should match the CPU limit, not the node's core count; GOMEMLIMIT (Go 1.19+) is a soft limit that trades CPU for heap stability.

env:
  - name: GOMAXPROCS
    value: "8"          # == CPU limit, not node cores
  - name: GOMEMLIMIT
    value: "1600MiB"    # ~90% of the container memory limit
  - name: GOGC
    value: "50"         # lower heap peaks at the cost of more GC cycles

Profiling in Production: pprof, trace, and the Right Numbers

func startDiagnostics(ctx context.Context, addr string) error {
	runtime.SetBlockProfileRate(10_000)  // sample ~1 in 10k blocking events
	runtime.SetMutexProfileFraction(100) // 1% of contended mutex events

	mux := http.NewServeMux()
	mux.HandleFunc("/debug/pprof/", pprof.Index)
	mux.HandleFunc("/debug/pprof/profile", pprof.Profile)
	mux.HandleFunc("/debug/pprof/trace", pprof.Trace)

	srv := &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
	go func() {
		<-ctx.Done()
		shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
		defer cancel()
		_ = srv.Shutdown(shutdownCtx)
	}()
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		return fmt.Errorf("diagnostics server: %w", err)
	}
	return nil
}

Serve this on a separate, unauthenticated-free port bound to loopback or a private interface — the default net/http/pprof import registers handlers on DefaultServeMux, which is a known information-disclosure finding. Block and mutex profiles are off by default; without SetBlockProfileRate and SetMutexProfileFraction, your "why is p99 bad" investigation sees nothing.

Use go tool trace for scheduler visibility: GC pauses, goroutine runnability, and syscall blocking. For a quick smoke test, GODEBUG=gctrace=1,schedtrace=1000 prints GC cycles and scheduler summaries to stderr every second.

Production Checklist for Concurrent Go Services

• Bound every fan-out with a pool or errgroup.SetLimit; the limit is a function of downstream capacity, not CPU count.
• Every `go` statement either owns a select on ctx.Done() or terminates provably.
• `defer cancel()` on every derived context; run go vet ./... in CI.
• Never close a channel with concurrent senders; signal via a dedicated quit channel.
• Preallocate slices whose size is known at call time.
• Profile with `alloc_objects` first, inuse_space second — not CPU profiles alone.
• Set `GOMEMLIMIT` below the container limit; set GOMAXPROCS to the CPU quota.
• Load-test with a degraded dependency (500 ms p99), not a healthy one — that is where unbounded concurrency fails.

Sources & Technical References

• The Go Programming Language — Effective Go: Concurrency: https://go.dev/doc/effective_go#concurrency
• The Go Memory Model (updated 2022): https://go.dev/ref/mem
• Diagnostics (official Go documentation): https://go.dev/doc/diagnostics
• Profiling Go Programs (Go Blog): https://go.dev/blog/pprof
• A Guide to the Go Garbage Collector (incl. GOMEMLIMIT): https://go.dev/doc/gc-guide
• runtime package reference (SetBlockProfileRate, SetMutexProfileFraction, MemProfileRate): https://pkg.go.dev/runtime
• runtime/trace package documentation: https://pkg.go.dev/runtime/trace
• golang.org/x/sync/errgroup (incl. SetLimit): https://pkg.go.dev/golang.org/x/sync/errgroup
• gRPC-Go — Quick start & basics: https://grpc.io/docs/languages/go/basics/
• google.golang.org/grpc/status package reference: https://pkg.go.dev/google.golang.org/grpc/status
• Go Blog — Contexts and Structs: https://go.dev/blog/context-and-structs
• Go Wiki — Code Review Comments: https://go.dev/wiki/CodeReviewComments
• R. D. Blumofe and C. E. Leiserson, "Scheduling Multithreaded Computations by Work Stealing," Journal of the ACM, 46(5), 1999: https://dl.acm.org/doi/10.1145/324133.324234
• R. Pike, Concurrency Is Not Parallelism (Heroku Waza, 2012): https://go.dev/talks/2012/waza.slide
• RFC 9113 — HTTP/2 (flow control and stream concurrency): https://www.rfc-editor.org/rfc/rfc9113