torch.profiler and nsys already give you per-kernel GPU time, so what does Graphsignal add? Comparable JSON snapshots, telemetry in the same document, GPU probes, overhead low enough to run in production, and a runner that wraps any engine.
A fair question about the open-source Graphsignal profiler: why use it instead of a torch.profiler Chrome trace or an nsys report? Both already contain per-kernel timing, and both can be analyzed outside a GUI.
They can, and on the kernel totals the three tools overlap. A script over a torch.profiler trace, or nsys stats --report cuda_gpu_kern_sum, gives you the same cumulative time per kernel that Graphsignal reports. nsys can time the kernels inside a CUDA graph with --cuda-graph-trace=node, and so can Graphsignal with the same flag on graphsignal-run. Graphsignal does all of that too.
The difference is how the data is delivered. A trace is a window you capture, write to a file, and process afterwards. Graphsignal is a sidecar that keeps running counters and serves them as one JSON document while the workload takes traffic:
graphsignal-run vllm serve <model> --port 8000curl -s http://127.0.0.1:18259/signalsThat gives you five things.
Each read of /signals has the same structure: the same metric names, the same profile frames sorted by cumulative time, the same units. Two runs, two flag settings, or two polls ten minutes apart can be compared as they are. There is no trace to parse and no events to aggregate before you can ask “did this change help?”. A trace answers that too, but only after you have written the script that turns two traces into two comparable tables.
A kernel trace tells you where GPU time went. It does not tell you that the GPU was throttling, that the engine’s queue was ten requests deep, that a worker had crashed with a traceback twenty minutes ago, or which driver and clocks the machine was running. Graphsignal puts all of that next to the kernel profile: NVML telemetry (utilization, memory, power, clocks, throttling, XID errors), the engine’s own Prometheus metrics (vLLM, SGLang, TensorRT-LLM), errors extracted from console output, and the host, process, and GPU configuration. With a trace, getting the same picture means three or four other tools and lining up their timestamps.
Once the profile names a kernel, the next question is which part of it. Graphsignal ships a single-header probe API you vendor into your engine or kernels. Register an instrument, record from host code or from inside a CUDA or HIP kernel, and the values appear in the same /signals document beside the built-in metrics. The record path is a few relaxed atomics, so probes are safe to leave in. This matters mostly if you are building an engine or writing kernels, which is what we do. A trace stops at the kernel boundary; probes go inside.
The default collection mode uses the CUPTI and ROCm activity APIs inside the workload and does everything else in the sidecar process. It needs no root and no GPU profiling privileges, and it is cheap enough to leave on under real traffic. That changes what you profile: not a ten-second capture you chose to take, but the workload under the load it really gets. The measured numbers are in the overhead guide.
torch.profiler needs the code wrapped in a profiler context, or an engine that exposes start and stop endpoints for it, and it only sees PyTorch. graphsignal-run wraps the launch command, so vLLM, SGLang, TensorRT-LLM, llama.cpp, or any CUDA or ROCm process works with no code changes and no imports. Engine-specific setup, like enabling the Prometheus endpoint, is handled by the runner.
For a timeline of one decode step, with kernels laid out on streams and host activity above them, use nsys or torch.profiler. That is what they are for, and Graphsignal does not replace them.
For the loop where you run the workload, put load on it, read where the time goes, change a flag or a kernel, and read again, the running counters are the better fit. That loop is increasingly run by an AI agent, and an agent works much better with a small JSON document it can poll and compare than with a two-gigabyte Chrome trace per iteration. The AI optimization guide walks through it.