gRPC in Production: Real Load Balancing over HTTP/2, Propagated Deadlines, Keepalive, and Evolving Protobuf Without Breaking Clients

A practical guide to what separates gRPC that works on a laptop from gRPC that survives production: why a ClusterIP Service leaves three pods at 90% CPU and nine at 4% because HTTP/2 distributes connections rather than requests, the four load balancing models (layer 4, L7 proxy, client-side with a headless Service and round_robin, and proxyless xDS) including the lazy DNS resolver that makes every scale-up invisible until you set MaxConnectionAge on the server, deadlines versus timeouts and how to propagate the remaining budget while reserving a margin (with the context.Background() trap and orphaned work), which of the 17 status codes to retry and why retrying DEADLINE_EXCEEDED amplifies an overload, declarative retry policy with retryThrottling as a circuit breaker, keepalive kept consistent across client and server so you do not trigger GOAWAY with ENHANCE_YOUR_CALM, real streaming backpressure and why raising the message limit to 100 MiB is an OOMKill waiting to happen, schema evolution with reserved, explicit presence in edition 2024 and buf breaking in CI, health checking with grpc.health.v1 and native Kubernetes probes, and the four-step shutdown sequence that removes the error spike from every deploy. With production-ready Go, Python, YAML, JSON and Protobuf, eight recurring failures, a production checklist, FAQ and glossary. Expanded edition, 11 September 2026: channel security with TLS and mTLS (RequireAndVerifyClientCert, certificate rotation without restarts via GetCertificate, and per-call credentials that gRPC refuses to send without TLS), the full error model (the real difference between INVALID_ARGUMENT, FAILED_PRECONDITION and UNAVAILABLE, the ABORTED / ALREADY_EXISTS pair, and rich google.rpc.Status details with ErrorInfo, BadRequest and RetryInfo, including the 8 KiB trailer limit), HTTP/2 windows and limits (why you get stuck at 100 streams, a channel pool versus raising max_concurrent_streams, the connection window that must exceed the stream window, and the 4 MiB message cap), gRPC-Web and why the browser cannot speak native gRPC (the Envoy filter, expose_headers with grpc-status, and the Connect alternative), and how a gRPC service is actually tested (in-memory bufconn, deadline and cancellation tests, buf breaking in CI, and percentiles with ghz).