Profiler Overhead

Graphsignal collects GPU activity inside the workload process and does everything else — analysis, statistics, NVML polling, the HTTP endpoint — in the sidecar profiler process. None of the modes below needs root or GPU profiling privileges.

There are three collection modes, and they cost different amounts. The short version: leave the default on, including in production; graph node tracing is the one to switch on for an investigation and off again; GPU probes are safe to ship in production code, and what you pay for them depends on how densely you record.

Which mode to use when

Kernels (default) — just graphsignal-run. Cumulative time per kernel, per CUDA graph, per transfer and synchronization kind, plus telemetry and engine metrics. This is what you leave running: on a GPU-bound engine it costs about as much as the benchmark’s own noise, and single-digit percent on an engine whose per-token host path is thin.

Graph node trace — --cuda-graph-trace node. Use it when the default mode puts all of the GPU time in cuda_graphs_nanoseconds and you need it broken down per kernel, which is the normal situation for engines that replay CUDA graphs in decode. It costs nothing measurable on a GPU-bound engine and up to roughly ten percent on a host-bound one, so run it for the investigation and go back to the default.

GPU probes — instrument the code. Reach for these when a kernel is named and the question is which part of it, or when the thing you want to measure is a concept in your code rather than a linker symbol.

Probes are built to stay in production code: the record path is lock-free — a few relaxed atomics, no allocation, no locks — and probes are inert when nothing reads them, so a build carrying them behaves the same whether or not a profiler is attached. What costs something is density, not their presence. A handful of instruments on request or iteration boundaries is not measurable. Instrumentation dense enough to answer “which part of this kernel” — hundreds to thousands of records per iteration, several per instruction per SM — costs a few percent; when you are that dense, take the numbers you publish from an unprobed build, because the instrumentation can outweigh the change you are measuring.

Measured

Measured on a single-GPU Blackwell host as relative change in decode time per token against the same workload run without graphsignal-run. Two LLM decode workloads, because one number is not a trend: an engine that replays a CUDA graph per token and is GPU-bound, and a llama.cpp server whose per-token host path is comparatively thin.

Mode	Graph-replaying engine, GPU-bound	llama.cpp server, thin host path
Kernels (default)	+0.1 % — inside the ±0.9 % run-to-run spread	+1.3 %
Graph node trace	+0.3 % — inside the ±0.9 % run-to-run spread	+8.1 % (spread ±5.5 %, so read it as “a few per cent to ten”)
GPU probes, dense instrumentation	+3.9 % recording a few hundred device instruments per kernel block; +4.8 % recording ~1400, several per instruction per SM	null — no instrumented build exists to measure

Both probe figures are deliberately heavy cases, chosen to bound the cost: production-shaped instrumentation — a few instruments per request or per decode step — did not move either workload out of its noise floor.

Method: the engine figures are the median of nine fresh processes per mode with the modes interleaved, 128 tokens at a 256-token prompt; the llama.cpp figures are the median of five fresh server starts per mode with the mode order rotated, three benchmark passes each at the same prompt. null means not measured.

Two cells are inside the noise floor rather than zero: on the GPU-bound engine both profiled modes fall within the ±0.9 % run-to-run spread, so the honest statement is that neither is measurably more expensive there, even though the ordering (unprofiled < default < node) held in every repetition.

Why the two workloads differ

Overhead tracks the rate at which the workload asks the driver to do things, not the size of the model. An engine that launches thousands of small kernels per second pays for each record; one that spends milliseconds inside each kernel, or replays a whole decode step as a single graph, has almost nothing to record. That is also why graph node tracing is nearly free in the first case and noticeable in the second: it replaces one record per graph replay with one record per kernel inside it, so its cost scales with how many kernels the graph contains and how often it runs.