Prometheus Metrics

Introduction

Besides the JSON /signals payload for agents, the profiler serves the same metrics in the Prometheus text exposition format at GET /metrics, on the same listener. Teams that already run Prometheus and Grafana can scrape a profiled workload like any other target, with no bridge in between: GPU and process telemetry, the engine’s own metrics, probes, and the kernel, CUDA graph, memcpy, and sync profiles.

The endpoint renders the profiler’s in-memory store on each request. The store holds cumulative snapshots, so nothing is reset by a scrape, and /signals reads stay unaffected.

Scrape it

Run the workload under graphsignal-run (see the Quick Start) and check the endpoint:

curl -s http://127.0.0.1:18259/metrics

Then add it to Prometheus as a target:

scrape_configs:
  - job_name: graphsignal
    scrape_interval: 5s
    static_configs:
      - targets: ['127.0.0.1:18259']

The endpoint binds to 127.0.0.1 by default. A Prometheus server on another machine or in another container needs the profiler started with --listen-host (for example --listen-host 0.0.0.0) — the endpoint is unauthenticated, so read Security and Privacy first and restrict access at the network level. The port is --listen-port (default 18259); see the CLI reference.

The endpoint lives only as long as the profiled workload. Once the workload exits, Prometheus marks the target down and its series go stale, as for any other target that stops.

How signals map

Profiler type	Prometheus families	Notes
gauge	<name> (gauge)	The latest value.
counter	<name>_total (counter)	The suffix is added unless the name already ends with _total, so an engine’s counter scraped by the profiler keeps its original name.
histogram	<name>_bucket{le}, <name>_count, <name>_sum (histogram); <name>_min, <name>_max (gauges)	_sum only when the source reports an exact sum; _min/_max only when the source keeps lifetime extremes (probes and the CUDA/ROCm libraries do).
profile	<name>_total, <name>_samples_total (counters)	One series per frame: cumulative value and sample count.

Names are made valid for Prometheus: any character outside [a-zA-Z0-9_:] becomes _ (user.hist → user_hist), and vLLM’s vllm: prefix is kept.

Labels

Every series carries its metric’s tags as labels, with dots and other invalid characters turned into _ (process.pid → process_pid, device.uuid → device_uuid). The run’s context tags — host.name, container.id, pod.uid, and every GRAPHSIGNAL_TAG_<NAME> (see Using Tags) — are added to every series as well; when a metric tag and a context tag map to the same label, the metric tag wins.

instance.id is left off the series on purpose: it changes on every run, and a new value on every series would start a fresh set of series with each restart. It is on one info gauge instead, together with the profiler version and the context labels:

graphsignal_profiler_info{host_name="gpu-node-1",version="1.1.2",instance_id="e3fb15db0053"} 1

Join it onto other series when you need the run identity, for example process_memory_usage_bytes * on(host_name) group_left(instance_id) graphsignal_profiler_info.

A tag whose label a series already uses for its own structure — le on a histogram, kernel or frame on a profile — is renamed exported_<label>, following Prometheus’s own convention.

Profiles in Grafana

A profile becomes labelled counters, one series per frame, so the kernel ranking a flame graph or graphsignal-top would show becomes a PromQL query. The frame labels depend on the profile:

Profile	Frame labels
cuda_kernels_nanoseconds, rocm_kernels_nanoseconds	kernel — the readable qualified function (vllm::rms_norm_kernel); symbol — the raw symbol, which keeps template instantiations apart; category (attention, gemm, norm, …); library (cuBLAS, nccl, …; absent when unknown)
cuda_graphs_nanoseconds	graph — the 16-hex structural signature of the CUDA graph
everything else (cuda_memcpy_nanoseconds, cuda_sync_nanoseconds, probe profiles)	frame — the frame name

The top 20 kernels by GPU time, in nanoseconds per second:

topk(20, rate(cuda_kernels_nanoseconds_total[1m]))

GPU time by kernel category:

sum by (category) (rate(cuda_kernels_nanoseconds_total[1m]))

Mean time per launch, from the sample counts:

rate(cuda_kernels_nanoseconds_total[1m]) / rate(cuda_kernels_nanoseconds_samples_total[1m])

In Grafana, the first query reads best as a bar gauge or a table (format as table, instant query, legend {{kernel}}), and the second as a stacked time series. There is no flame graph view: the profiles are flat rankings by design.

Engines that replay CUDA graphs in decode put that time in cuda_graphs_nanoseconds, not in cuda_kernels_nanoseconds, unless the profiler runs with --cuda-graph-trace node — see CUDA graphs and the kernel profile.

Histograms

Histograms work with histogram_quantile over the bucket rates:

histogram_quantile(0.95, sum by (le) (rate(vllm:e2e_request_latency_seconds_bucket[5m])))

Engine histograms keep the engine’s own le bounds. Histograms recorded by GPU probes and the CUDA/ROCm libraries use fixed log-linear bins (exact for 0–3, then 4 per power of two), reported as each bin’s largest value, so every le bound is exact and the quantile resolution is the same ≤25% that p50/p95 have in /signals. Only bins that have received observations are exposed, so new le buckets can appear over time.

A source that reports totals without buckets — a Prometheus summary scraped from the engine — is exposed as a histogram with only the +Inf bucket, _count, and _sum; averages work, quantiles do not. <name>_min and <name>_max are lifetime extremes, not windowed ones.

Caveats

• Frame eviction. A profile keeps its top 250 frames. A cold frame that drops out and later comes back reads as a counter reset to rate(); the hot frames a leaderboard shows are not affected.
• Metric expiry. A metric not updated for 10 minutes — a finished worker’s series, for example — disappears from the endpoint, and its Prometheus series go stale.
• Bucket resolution. Probe and CUDA/ROCm library histograms have bins up to 25% wide; engine histograms have the engine’s own bounds. An engine histogram’s observations above its largest finite bound are counted in that bound’s bucket.
• Units. Profile and probe names keep their units as in /signals (cuda_kernels_nanoseconds_total is in nanoseconds), rather than Prometheus’s base unit of seconds; promtool check metrics notes this, and the colons in vllm: names, as lint warnings.