Secrets Management in Production: From .env to Workload Identity, Zero-Downtime Rotation and a Leak Runbook

A practical guide to why teams lose keys to operations rather than cryptography, and how to stop: what counts as a secret and what does not, why .env breaks down in production, the three detection layers that keep secrets out of the repository (pre-commit with gitleaks, full-history scanning in CI and push protection), and SOPS with age when config has to live in Git. Covers the shift that removes the most risk —preferring identity over secrets with IRSA, Workload Identity and OIDC federation in CI—, including the IAM trust policy and the sub wildcard that turns a key-free architecture into an open door; dynamic PostgreSQL credentials with Vault and its two operational traps (pool renewal and orphaned roles); the real difference between delivering a secret via environment variable or mounted file (/proc, subprocess inheritance, and why env never updates on rotation); settings with pydantic-settings and SecretStr over secrets_dir; a RotatingSecret wrapper that re-reads the file when the kubelet updates it; Kubernetes with etcd encryption at rest, per-namespace RBAC and syncing via External Secrets Operator (ClusterSecretStore, ExternalSecret and volume mounting); zero-downtime rotation with the two-credential pattern and the per-key_id metric that tells you when revoking is safe; secret redaction in logs with structlog; a leak runbook that starts with revoking rather than rotating; eight recurring mistakes and a production checklist. With production-ready Python, YAML, JSON and Bash. It also includes a deep dive into the blast radius of each credential, OIDC federation in CI/CD with the July 2026 immutable claim change, secrets in container images with BuildKit mounts, Terraform state and the 1.11 write-only arguments, Vault dynamic credentials and their two operational traps, KMS envelope encryption, the mobile and web client case, and the metrics that tell you whether the programme is improving.