Prometheus Metrics in Production: Histograms, Percentiles That Do Not Lie, Cardinality, and Burn-Rate SLO Alerting

A practical guide to the observability pillar that gets instrumented the most and understood the least: why average latency hides exactly what you need to see, how histogram_quantile actually works under the hood (linear interpolation inside the bucket, and the highest-finite-bucket ceiling that pins your p99 at 10s), the aggregation rule that decides whether your dashboard is correct or decorative (rate first, sum by (le) second, quantile last), choosing buckets at your service scale, native histograms now stable in Prometheus 3.8 and how to migrate without losing history using always_scrape_classic_histograms, the cardinality arithmetic that explains why one new label multiplies your series by a thousand and the defenses that contain it (sample_limit, label_limit, metric_relabel_configs), the RED and USE methods, recording rules with the right naming convention, and multi-window multi-burn-rate SLO alerts (14.4x over 1h, 6x over 6h, 1x over 3d) that replace arbitrary latency thresholds with something you can defend. Includes a complete FastAPI instrumentation middleware (with the multiprocess detail that silently breaks metrics under Gunicorn), annotated PromQL, production-ready YAML rules, exemplars to jump from a percentile to the exact trace, eight recurring mistakes, a production checklist, and an FAQ.