Find the resource behind the goroutine
Goroutines are cheap; the database connections, file descriptors, remote quotas, and memory they consume are not. A service-wide limit can prevent collapse, but it lets one slow dependency starve unrelated traffic. Place limits near each scarce resource and add a broader admission limit only when the process needs an overall safety envelope.
select {
case limiter <- struct{}{}:
defer func() { <-limiter }()
case <-ctx.Done():
return context.Cause(ctx)
}
return callDependency(ctx)Waiting consumes the caller's budget
Semaphore acquisition must honor cancellation. Record queue duration separately from execution time so a saturated limiter is visible. Fairness matters when requests have different costs: a simple token channel may be sufficient for uniform calls, while weighted work needs a policy that prevents large requests from waiting forever.
Align limits with downstream pools. Allowing 500 handlers to compete for 20 database connections creates an invisible second queue inside the driver. A small application-side limit can reject or shed earlier, where the service still knows request priority. Avoid independent retry storms on top of these queues.
Capacity is a feedback loop
Track in-flight work, wait time, rejections, dependency latency, and success rate. Increase a limit only when the resource has headroom and the workload benefits. The goal is not maximum concurrency; it is stable useful throughput with bounded latency.
Today's implementation checklist
Before changing production code, write down the resource being protected, the failure signal a caller will see, and the metric that will prove improvement. Prefer a small reversible change. Exercise cancellation and overload paths, then review the resulting profile or trace beside the baseline. The discipline is intentionally plain: make ownership visible, put a bound on waiting, and preserve enough evidence to explain the outcome tomorrow.