Continuous Profiling in Production: Flame Graphs, pprof, py-spy, eBPF, and OpenTelemetry's Fourth Signal
A practical guide to the observability signal almost nobody has instrumented, and the one that answers the question traces leave half-finished: the trace tells you which span took 800 ms, the profile tells you which line ate those 800 ms. Covers the statistics of sampling (how many samples you need before a 5% figure means anything, and why 99 Hz instead of 100), how to read a flame graph without fooling yourself (the X axis is not time, width is samples, and plateaus are self time), Go with pprof wired up correctly on a separate admin listener — not the public DefaultServeMux — the six profile types and the question each one answers, SetMutexProfileFraction and SetBlockProfileRate with their real cost, pprof labels for attributing CPU per tenant plus the detail that only the CPU profile carries them, the alloc_space versus inuse_space confusion that sends people off optimizing the wrong function, and Go 1.25's FlightRecorder for the cases where the problem isn't CPU at all but that nothing was running. In Python, py-spy attaching by PID without touching your code, CAP_SYS_PTRACE and the Yama ptrace_scope that makes attaching fail inside containers, why cProfile doesn't belong in production, memray for native memory, and the new standard-library sampling profiler arriving in Python 3.15 (Tachyon) with its wall, cpu, gil and exception modes, differential flame graphs, and the requirement that profiler and target run the exact same Python minor version. In Node.js, --cpu-prof and a programmatic inspector session that captures 30 seconds without restarting the process. Then the jump from ad-hoc profiling to continuous profiling with Pyroscope and Grafana Alloy: an SDK configuration whose labels include the deploy SHA — the thing that makes a regression attributable — eBPF profiling with no application changes and its two real limits (frame pointers, which Fedora 38 and Ubuntu 24.04 turned back on by default, and interpreted languages that need their own unwinders), measured overhead of 2-5% CPU and under 50 MB per pod, and the cardinality explosion that behaves exactly like it does in Prometheus. It closes with the honest state of the OpenTelemetry profiles signal, in public alpha since March 2026, with lossless round-tripping against pprof and roughly 40% smaller wire size thanks to a shared string dictionary, how to link traces and profiles today with span profiles, and the memory chapter where the heap profile says nothing because the real problem is fragmentation, native allocations, or glibc arenas. With production-ready code in Go, Python, JavaScript, YAML, and shell, eight recurring mistakes, a production checklist, FAQ, and glossary. Expanded edition of 16 September 2026: the JVM in depth, with the safepoint bias that makes classic Java profilers lie, async-profiler 4.3 with AsyncGetCallTrace, wall-clock sampling and the two startup flags without which you land back on the problem you were avoiding, and JFR in ring mode under 1% overhead with the kernel CPU-clock sampling that arrived with JEP 509 in JDK 25; off-CPU profiling with the sched_switch tracepoint and offcputime, why the X axis is microseconds blocked rather than samples, and a script that merges on-CPU and off-CPU stacks into a single wall-clock flame graph; symbols and unwinding, with frame pointers versus DWARF and stack deltas, Build ID and debuginfod, the -ldflags="-s -w" that ruins Go profiles, and the per-runtime unwinders that force you to upgrade the profiler when you upgrade the interpreter; CI profiling turned into a real guardrail with interleaved comparison, benchstat and the differential pprof with -base; the cost of continuous profiling with the four levers that move the bill and Pyroscope 2.0; and a full case of a 38% CPU jump introduced by a deploy, solved in eleven minutes thanks to the version label.
Verificando acceso...