Guide
Premium
Intermediate
Real Projects

Continuous Profiling in Production: Flame Graphs, pprof, py-spy, eBPF, and OpenTelemetry's Fourth Signal

A practical guide to the observability signal almost nobody has instrumented, and the one that answers the question traces leave half-finished: the trace tells you which span took 800 ms, the profile tells you which line ate those 800 ms. Covers the statistics of sampling (how many samples you need before a 5% figure means anything, and why 99 Hz instead of 100), how to read a flame graph without fooling yourself (the X axis is not time, width is samples, and plateaus are self time), Go with pprof wired up correctly on a separate admin listener — not the public DefaultServeMux — the six profile types and the question each one answers, SetMutexProfileFraction and SetBlockProfileRate with their real cost, pprof labels for attributing CPU per tenant plus the detail that only the CPU profile carries them, the alloc_space versus inuse_space confusion that sends people off optimizing the wrong function, and Go 1.25's FlightRecorder for the cases where the problem isn't CPU at all but that nothing was running. In Python, py-spy attaching by PID without touching your code, CAP_SYS_PTRACE and the Yama ptrace_scope that makes attaching fail inside containers, why cProfile doesn't belong in production, memray for native memory, and the new standard-library sampling profiler arriving in Python 3.15 (Tachyon) with its wall, cpu, gil and exception modes, differential flame graphs, and the requirement that profiler and target run the exact same Python minor version. In Node.js, --cpu-prof and a programmatic inspector session that captures 30 seconds without restarting the process. Then the jump from ad-hoc profiling to continuous profiling with Pyroscope and Grafana Alloy: an SDK configuration whose labels include the deploy SHA — the thing that makes a regression attributable — eBPF profiling with no application changes and its two real limits (frame pointers, which Fedora 38 and Ubuntu 24.04 turned back on by default, and interpreted languages that need their own unwinders), measured overhead of 2-5% CPU and under 50 MB per pod, and the cardinality explosion that behaves exactly like it does in Prometheus. It closes with the honest state of the OpenTelemetry profiles signal, in public alpha since March 2026, with lossless round-tripping against pprof and roughly 40% smaller wire size thanks to a shared string dictionary, how to link traces and profiles today with span profiles, and the memory chapter where the heap profile says nothing because the real problem is fragmentation, native allocations, or glibc arenas. With production-ready code in Go, Python, JavaScript, YAML, and shell, eight recurring mistakes, a production checklist, FAQ, and glossary. Expanded edition of 16 September 2026: the JVM in depth, with the safepoint bias that makes classic Java profilers lie, async-profiler 4.3 with AsyncGetCallTrace, wall-clock sampling and the two startup flags without which you land back on the problem you were avoiding, and JFR in ring mode under 1% overhead with the kernel CPU-clock sampling that arrived with JEP 509 in JDK 25; off-CPU profiling with the sched_switch tracepoint and offcputime, why the X axis is microseconds blocked rather than samples, and a script that merges on-CPU and off-CPU stacks into a single wall-clock flame graph; symbols and unwinding, with frame pointers versus DWARF and stack deltas, Build ID and debuginfod, the -ldflags="-s -w" that ruins Go profiles, and the per-runtime unwinders that force you to upgrade the profiler when you upgrade the interpreter; CI profiling turned into a real guardrail with interleaved comparison, benchstat and the differential pprof with -base; the cost of continuous profiling with the four levers that move the bill and Pyroscope 2.0; and a full case of a 38% CPU jump introduced by a deploy, solved in eleven minutes thanks to the version label.

49 minutes read
0 views

Verificando acceso...

Loading comments...

Related Resources

Guía
PREMIUM

API Versioning and Contract Testing: Safe Changes, OpenAPI in CI, Pact, and Retiring Endpoints with Sunset

A practical guide to changing your APIs without breaking the people who consume them: why compatibility rules invert between the request and the response —and why widening a returned enum breaks clients even though you are "only adding"—, the tolerant reader pattern in Pydantic with an escape hatch and a metric, OpenAPI generated from the code and diffed on every pull request with oasdiff (including the git diff --exit-code step without which the whole check is theatre), consumer-driven contract testing with Pact: type matchers instead of literal values, well-designed provider states, version selection with deployed_or_released, and the gate that actually makes it safe, can-i-deploy paired with record-deployment. It also covers rolling this out without stopping the factory using pending and WIP pacts, bi-directional contracts when the provider is a third party, the three versioning strategies with their real operational costs, why versioning the whole API for a single endpoint guarantees nobody migrates, retiring versions with the Deprecation (RFC 9745) and Sunset (RFC 8594) headers plus the migration Link, the per-consumer metric without which no sunset date is ever met, brownouts returning 410 Gone before the final shutdown, and the BACKWARD, FORWARD and FULL compatibility modes for event schemas. With production-ready code in Python, YAML, Bash and PromQL, eight recurring mistakes, a production checklist, FAQ and glossary. It also extends the contract beyond the happy path: errors with RFC 9457 (problem+json), cursor pagination and defaults as part of the contract, the expand/contract pattern for renaming a field across database and API with no maintenance window, governance with Spectral, buf breaking for gRPC and Protobuf, GraphQL schema evolution with @deprecated and real per-field usage, and semantic versioning of generated SDKs. It also covers the contracts that never show up in the schema and break just as hard: outbound webhooks with the version pinned on the subscription, HMAC signing with a timestamp window and key rotation, Idempotency-Key for safe POST retries, the RateLimit and RateLimit-Policy headers —and why lowering a limit is a breaking change—, and OAuth scopes, the blind spot no OpenAPI diff will ever catch.

Guía
PREMIUM

asyncio in Production: Never Block the Event Loop — TaskGroups, Cancellation and Bounded Concurrency

A practical asyncio guide for Python services in production: why blocking the event loop degrades the whole process without raising a single exception, how to catch it by measuring loop lag and with Python 3.14 introspection, structured concurrency with TaskGroup and handling ExceptionGroup via except*, the task the garbage collector makes vanish, timeouts with a deadline budget propagated across services, correct cancellation with cleanup and shield, bounded concurrency with semaphores and backpressured queues, synchronization primitives, and what changes with eager tasks, python -m asyncio pstree and free-threading. With production-ready code and a deployment checklist.

Guía
PREMIUM

Cache-Aside in Production: TTLs, Invalidation, and How to Prevent Cache Stampedes

The complete guide to the cache-aside pattern with Redis: jittered TTLs, correct invalidation, and the three defenses against cache stampedes (distributed lock, single-flight, and XFetch). Expanded with stale-while-revalidate, fail-open and circuit breakers, two-tier caching with RESP3 invalidation, delayed double delete and CDC, hot keys, eviction and memory management, observability with Prometheus, testing, choosing an engine (Redis, Valkey, Memcached), and a complete TypeScript implementation. With production-ready code in Python and TypeScript.