SLOs and Error Budgets in Production: SLIs That Measure What Hurts, Multi-Window Burn Rate, and Alerts Nobody Silences

A practical guide to alerting on consequences instead of causes: how to pick SLIs that break when users suffer and why the measurement point (application, load balancer or RUM) changes the answer, histogram buckets aligned with the SLO threshold, the PromQL mistake that invalidates half the latency SLOs out there — histogram_quantile versus the fraction of events under the threshold — and native histograms, stable since Prometheus 3.8, with histogram_fraction, per-window recording rules with the 'or ... * 0' that tells zero errors apart from no data, burn rate derived from the formula with Google's table (14.4 / 6 / 3 / 1) and why two windows are required, copy-ready multi-window multi-burn-rate alerts, automatic generation with Sloth and OpenSLO verified in CI with a diff, a Python script that blocks the deploy when less than 25% of the budget remains, and the error budget policy that turns an SLO into decisions rather than decoration. With eight recurring mistakes, a production checklist, FAQ and glossary. Production-ready code in Python, PromQL and YAML.

34 minutes read

Josué Puig

2026-09-17

1 views

Verificando acceso...

Loading comments...

Related Resources

Guía

PREMIUM

API Versioning and Contract Testing: Safe Changes, OpenAPI in CI, Pact, and Retiring Endpoints with Sunset

A practical guide to changing your APIs without breaking the people who consume them: why compatibility rules invert between the request and the response —and why widening a returned enum breaks clients even though you are "only adding"—, the tolerant reader pattern in Pydantic with an escape hatch and a metric, OpenAPI generated from the code and diffed on every pull request with oasdiff (including the git diff --exit-code step without which the whole check is theatre), consumer-driven contract testing with Pact: type matchers instead of literal values, well-designed provider states, version selection with deployed_or_released, and the gate that actually makes it safe, can-i-deploy paired with record-deployment. It also covers rolling this out without stopping the factory using pending and WIP pacts, bi-directional contracts when the provider is a third party, the three versioning strategies with their real operational costs, why versioning the whole API for a single endpoint guarantees nobody migrates, retiring versions with the Deprecation (RFC 9745) and Sunset (RFC 8594) headers plus the migration Link, the per-consumer metric without which no sunset date is ever met, brownouts returning 410 Gone before the final shutdown, and the BACKWARD, FORWARD and FULL compatibility modes for event schemas. With production-ready code in Python, YAML, Bash and PromQL, eight recurring mistakes, a production checklist, FAQ and glossary. It also extends the contract beyond the happy path: errors with RFC 9457 (problem+json), cursor pagination and defaults as part of the contract, the expand/contract pattern for renaming a field across database and API with no maintenance window, governance with Spectral, buf breaking for gRPC and Protobuf, GraphQL schema evolution with @deprecated and real per-field usage, and semantic versioning of generated SDKs. It also covers the contracts that never show up in the schema and break just as hard: outbound webhooks with the version pinned on the subscription, HMAC signing with a timestamp window and key rotation, Idempotency-Key for safe POST retries, the RateLimit and RateLimit-Policy headers —and why lowering a limit is a breaking change—, and OAuth scopes, the blind spot no OpenAPI diff will ever catch.

Guía

PREMIUM

asyncio in Production: Never Block the Event Loop — TaskGroups, Cancellation and Bounded Concurrency

A practical asyncio guide for Python services in production: why blocking the event loop degrades the whole process without raising a single exception, how to catch it by measuring loop lag and with Python 3.14 introspection, structured concurrency with TaskGroup and handling ExceptionGroup via except*, the task the garbage collector makes vanish, timeouts with a deadline budget propagated across services, correct cancellation with cleanup and shield, bounded concurrency with semaphores and backpressured queues, synchronization primitives, and what changes with eager tasks, python -m asyncio pstree and free-threading. With production-ready code and a deployment checklist.

Guía

PREMIUM

Cache-Aside in Production: TTLs, Invalidation, and How to Prevent Cache Stampedes

The complete guide to the cache-aside pattern with Redis: jittered TTLs, correct invalidation, and the three defenses against cache stampedes (distributed lock, single-flight, and XFetch). Expanded with stale-while-revalidate, fail-open and circuit breakers, two-tier caching with RESP3 invalidation, delayed double delete and CDC, hot keys, eviction and memory management, observability with Prometheus, testing, choosing an engine (Redis, Valkey, Memcached), and a complete TypeScript implementation. With production-ready code in Python and TypeScript.