Profiler CLI

Graphsignal observes your workload from a sidecar process — the profiler. It runs out-of-process, never inside your workload. graphsignal-run launches a workload with the profiler attached.

Install the CLI as an isolated uv tool so it doesn’t pollute the workload environment:

UV_TOOL_BIN_DIR=/usr/local/bin uv tool install 'graphsignal[cu12]'   # CUDA 12.x
# or
UV_TOOL_BIN_DIR=/usr/local/bin uv tool install 'graphsignal[cu13]'   # CUDA 13.x

UV_TOOL_BIN_DIR=/usr/local/bin puts graphsignal-run in a directory that is already on PATH for every shell, including non-interactive scripts and containers. pip install 'graphsignal[cu12]' into the workload environment works too.

graphsignal-run

Wrap any launch command. graphsignal-run starts the profiler sidecar, enables GPU profiling, and launches your workload so process managers (init systems, container runtimes, etc.) see only the workload.

graphsignal-run <command> [args...]

Examples:

graphsignal-run vllm serve <model> --port 8001
graphsignal-run sglang serve --model-path <model>
graphsignal-run python -m sglang.launch_server --model-path <model>
graphsignal-run trtllm-serve <model> --port 8000
graphsignal-run --metrics-port 8000 trtllm-serve <model> --port 8000
graphsignal-run python myapp.py
graphsignal-run app.py

Options (must precede the command):

--version — Print the profiler version and exit.
--metrics-port PORT — Port to scrape the workload’s Prometheus /metrics endpoint on. Overrides the port derived from the engine’s --port flag or its default (e.g. 8000 for vLLM/TensorRT-LLM, 30000 for SGLang). Use this when metrics are exposed on a different port than the HTTP server. Not forwarded to the workload.
--listen-host HOST — Host to bind the /signals endpoint to (default 127.0.0.1, loopback only). Set e.g. 0.0.0.0 to expose the endpoint for remote access — the endpoint is unauthenticated, so anything that can reach that address can read it. Not forwarded to the workload.
--listen-port PORT — Port for the /signals endpoint (default 18259). Not forwarded to the workload.
--cuda-graph-trace {graph|node} — Granularity for CUDA graph launches (default graph). With graph, each graph replay is timed as a whole and reported in cuda_graphs_nanoseconds; the kernels inside it are not timed individually. With node, those kernels are timed individually and reported in cuda_kernels_nanoseconds by symbol, and cuda_graphs_nanoseconds stays empty. Use node to rank the kernels of an engine that replays CUDA graphs in decode (vLLM, SGLang, TensorRT-LLM, llama.cpp); it needs no extra privileges, but CUPTI does more work per replay, so prefer the default for long-running deployments. Ignored on ROCm, where rocprofiler-sdk reports every dispatch individually. Not forwarded to the workload.

Behavior:

Detects the engine from your command (vLLM, SGLang, TensorRT-LLM, or a generic fallback).
Scrapes Prometheus metrics from one known endpoint — http://127.0.0.1:<port>/metrics, or /prometheus/metrics for TensorRT-LLM — with the port resolved from --metrics-port, the engine’s --port, or the engine default. It never probes the workload’s other listening sockets; connecting to them blindly can corrupt internal IPC.
Collects GPU activity via CUPTI (or ROCm’s rocprofiler-sdk) as soon as the workload starts using the GPU.
Passes the workload command through byte-for-byte, with two metric-related adjustments: the SGLang launcher adds --enable-metrics (its Prometheus endpoint is off by default) and the vLLM launcher removes --disable-log-stats.
Traces CUDA graphs at graph granularity by default: a replay is one timing, and its kernels do not appear in the kernel profile. --cuda-graph-trace node switches to per-kernel granularity for the same graphs. The cuda_graph_trace_mode gauge in the payload always says which granularity produced it.
Serves everything measured at http://127.0.0.1:<listen-port>/signals while the workload runs.
Exits with the workload’s exact status.

graphsignal-top

A top-like terminal view of a workload running under graphsignal-run. It polls the profiler’s /signals endpoint and renders the live state: GPU telemetry with a utilization sparkline, engine throughput and latency, a GPU-time breakdown, and a table of kernels ranked by time with human-readable names.

graphsignal-top [options]

Examples:

graphsignal-top                                   # the local workload
graphsignal-top --host gpu-box --port 18259       # a remote one (its profiler needs --listen-host)
graphsignal-top --once                            # print one frame and exit

Options:

--host HOST — Host of the /signals endpoint (default 127.0.0.1). For a remote workload, the host its graphsignal-run was bound to with --listen-host.
--port PORT — Port of the /signals endpoint (default 18259).
--interval SEC — Seconds between polls (default 1).
--window SEC — Seconds over which rates, the GPU-time breakdown and the kernel ranking are computed (default 5). Longer windows smooth bursty workloads; shorter ones react faster.
--once — Print one frame to stdout and exit instead of the interactive view. Polls twice, --interval apart, so the frame carries real rates. Suitable for scripts, logs and agents.
--no-color — Disable colors. NO_COLOR in the environment does the same.
-V, --version — Print the profiler version and exit.
-h, --help — Show the usage and exit.

Screen, top to bottom:

Header — endpoint, the workload’s command line, uptime, and the poll status (live, paused, or the reason the endpoint is unreachable; the view keeps retrying). The second line names the pid, host and profiler version, and the CUDA graph trace mode: in graph mode the kernels replayed from CUDA graphs appear under the graphs tab rather than as kernels (see --cuda-graph-trace).
GPU — one block per device from the NVML telemetry: power, temperature, SM clock, PCIe traffic, XID errors and throttling on the title line; utilization over time as a sparkline (samples taken while throttled are highlighted); memory used as a percent bar. When the terminal is too short for the table, each device collapses to its title line.
Engine — from the scraped Prometheus metrics of vLLM, SGLang or TensorRT-LLM: running and waiting requests, KV-cache usage, prefix-cache hit rate, prompt and generation tokens per second over time, and p50/p95 of TTFT, time per output token, and end-to-end latency over the window. Hidden when no engine metrics are scraped.
GPU time — the share of a GPU-second spent in kernels, graph replays, memory copies and memsets over the window, plus host synchronization time and the idle remainder. Multi-GPU hosts average across devices.
Table — one row per kernel (or graph, transfer kind, or sync type, by tab): share of GPU time, milliseconds per second, calls per second, average duration, category and name. Kernel names come from a built-in taxonomy — attention, gemm, moe, norm, rotary, activation, kv cache, collective, sampling, elementwise, reduction, copy — with the demangled function and its library beside them; unrecognized kernels keep their demangled name under other. Nothing is hidden: the details view always has the full raw symbol. The memcpy tab lists memsets alongside the transfers.
Errors — the newest warning or error from the profiler, including exceptions extracted from the engine’s console; e lists them all.

Keys:

Key	Action
`k` `g` `m` `s`	Kernels, graphs, memcpy, sync tabs (`tab` cycles those with data)
`↑` `↓` `pgup` `pgdn`	Select a row
`⏎`	Details of the selected row: full symbol, library, totals since start, per-process breakdown
`p`	Pause polling
`e`	All recent errors
`?`	Help
`q`	Quit

The view reads the same /signals payload an agent would: every number on screen is a value from the payload or a difference between two consecutive payloads. It never connects to the workload itself.

Environment variables

The profiler reads its configuration from environment variables. Set these before invoking graphsignal-run.

Variable	Purpose
`GRAPHSIGNAL_LISTEN_HOST`	Host to bind the `/signals` endpoint to (same as `--listen-host`; default `127.0.0.1`).
`GRAPHSIGNAL_LISTEN_PORT`	Port for the `/signals` endpoint (same as `--listen-port`).
`GRAPHSIGNAL_TAG_<KEY>=<value>`	Arbitrary tag attached to all signals (e.g. `GRAPHSIGNAL_TAG_DEPLOYMENT=us-prod`).
`GRAPHSIGNAL_CUDA_GRAPH_TRACE`	`graph` (default) or `node` — CUDA graph tracing granularity, same as `--cuda-graph-trace`. The flag wins over this variable. An unrecognized value falls back to `graph` and is noted under `GRAPHSIGNAL_DEBUG=1`.
`GRAPHSIGNAL_DEBUG`	Set to `1` to log the profiler’s own diagnostics to stderr.
`GRAPHSIGNAL_API_KEY` (optional)	Enables the production feedback loop: signals are uploaded to Graphsignal. Without it, no profiling data is ever uploaded.
`GRAPHSIGNAL_API_BASE`	Override the upload endpoint (defaults to `https://api.graphsignal.com`).
`GRAPHSIGNAL_DISABLE_VERSION_CHECK`	Set to `1` to turn off the once-per-run check for a newer release (see Security and Privacy).