Signals API

Graphsignal exposes profiling data over HTTP in two places, and an AI agent optimizing a workload reads one or both:

	Local endpoint	Graphsignal API
Host	`http://127.0.0.1:18259` on the profiled machine	`https://api.graphsignal.com`
Authentication	none — loopback only	`X-API-Key`
Available	while the workload runs	after upload, for the retention period
Accumulates from	the instance start	a `start` the caller names

Reach for the local endpoint when the agent runs beside the workload — during development, in a container next to the engine, over an SSH tunnel. Reach for the Graphsignal API when it does not: production runs the agent cannot sit next to, comparisons across machines, or anything read after the workload exited.

Both answer the same payload shape, so an agent that can read one can read the other.

Local endpoint

The profiler serves this itself, with no API key and nothing uploaded anywhere. It is available whenever a workload runs under graphsignal-run.

The endpoint lives only as long as the profiled workload. Read it while the process runs; once the workload exits there is nothing listening, and the last errors are in its console output.

It binds to 127.0.0.1 by default, so it is not reachable from off the machine — see Security and Privacy before binding elsewhere with --listen-host.

GET /signals

The current snapshot. No parameters.

curl -s http://127.0.0.1:18259/signals

GET /health

{"status": "ok"} once the watcher is up. Useful for waiting on the profiler before the first read.

curl -s http://127.0.0.1:18259/health

Graphsignal API

When the profiler runs with GRAPHSIGNAL_API_KEY set, it uploads its signals to Graphsignal. These endpoints read them back — typically from an agent closing the production feedback loop.

Base URL: https://api.graphsignal.com

Authenticate with the API key in a request header:

X-API-Key: <your-api-key>

The same key the profiler uploads with reads its project’s signals back.

Common error responses:

401 Unauthorized: Invalid or missing credentials.
400 Bad Request: Missing or invalid query parameters.

GET /api/v1/instances

List the instances that have reported. Each profiler run is one instance, identified by its instance.id tag. This is how an agent finds what to read — the local endpoint needs no equivalent, because there is exactly one instance behind it.

Query parameters:

start (optional, integer): Start of time range in epoch seconds. Defaults to 24 hours before end.
end (optional, integer): End of time range in epoch seconds. Defaults to now.

Example request:

curl -s -H "X-API-Key: $GRAPHSIGNAL_API_KEY" \
  "https://api.graphsignal.com/api/v1/instances"

Response:

{
  "instances": [
    {
      "instance_id": "e3fb15db0053",
      "resource_id": "e057d0a0ed4a",
      "tags": {
        "instance.id": "e3fb15db0053",
        "host.name": "gpu-node-1",
        "process.pid": "28172",
        "api_key.name": "default",
        "project.name": "default"
      },
      "attributes": {
        "process.command_line": "vllm serve <model> --port 8001"
      },
      "first_seen_ts": 1787906183,
      "last_seen_ts": 1787906195
    }
  ]
}

tags carries everything the run was tagged with, including GRAPHSIGNAL_TAG_* values.

GET /api/v1/signals

One instance’s snapshot, reconstructed from the uploaded data. Two differences from the local endpoint: the top block is platform (the server’s version) instead of profiler, and the accumulation base is the caller’s start rather than the instance start — a production instance can outlive any retention, so the caller names the window.

Query parameters:

instance_id (required, string): The instance to read. Take it from /api/v1/instances, or from the local payload’s context.
start (required, integer): Epoch seconds. Counters, histograms and profiles accumulate from this instant (exclusive) to the snapshot instant. To watch an instance over time, pass the previous snapshot’s payload_ns / 1e9 here — consecutive windows tile without double counting.
end (optional, integer): Snapshot instant in epoch seconds, inclusive. Defaults to now.

Unknown instance_id returns 404 Not Found; a missing start or an inverted window returns 400 Bad Request. A start older than the data retention answers no metrics.

Example request:

curl -s -H "X-API-Key: $GRAPHSIGNAL_API_KEY" \
  "https://api.graphsignal.com/api/v1/signals?instance_id=e3fb15db0053&start=1787902595"

The payload

Both /signals endpoints answer this shape. The local one tops it with profiler, the Graphsignal one with platform; everything below that is identical.

{
  "profiler": {"version": "1.0.0"},
  "payload_ns": 1787906195000000000,
  "start_ns": 1787902595000000000,
  "context": {"instance.id": "e3fb15db0053", "host.name": "gpu-node-1"},
  "metrics": [
    {"name": "gpu_utilization_percent", "type": "gauge",
     "tags": {"device.uuid": "GPU-..."},
     "stats": {"value": 88.0},
     "updated_ns": 1787906195000000000},
    {"name": "cuda_kernels_nanoseconds", "type": "profile",
     "tags": {"process.pid": "28172"},
     "stats": {"frames": [
       {"name": "<kernel symbol>", "value": 8100000000, "samples": 3200},
       {"name": "<another kernel symbol>", "value": 550000000, "samples": 410}]},
     "updated_ns": 1787906195000000000},
    {"name": "vllm:e2e_request_latency_seconds", "type": "histogram",
     "tags": {"process.pid": "28172"},
     "stats": {"count": 1204, "sum": 505.68, "min": null, "max": null,
               "mean": 0.42, "p50": 0.3, "p95": 1.5},
     "updated_ns": 1787906195000000000},
    {"name": "cuda_memcpy_bytes", "type": "counter",
     "tags": {"kind": "host_to_device", "process.pid": "28172"},
     "stats": {"total": 1048576},
     "updated_ns": 1787906195000000000},
    {"name": "cuda_graph_trace_mode", "type": "gauge",
     "tags": {"process.pid": "28172"},
     "stats": {"value": 0.0},
     "updated_ns": 1787906195000000000}
  ],
  "errors": [
    {"level": "error", "message": "CUDA out of memory ...",
     "exception": "Traceback ...", "log_ns": 1787906190000000000}
  ],
  "resources": [
    {"kind": "host", "tags": {}, "attributes": {"platform.name": "Linux"},
     "first_seen_ts": 1787906183, "last_seen_ts": 1787906195},
    {"kind": "process", "tags": {"process.pid": "28172"},
     "attributes": {"process.command_line": "..."},
     "first_seen_ts": 1787906183, "last_seen_ts": 1787906195},
    {"kind": "device", "tags": {"device.uuid": "GPU-..."},
     "attributes": {"device.name": "NVIDIA ..."},
     "first_seen_ts": 1787906183, "last_seen_ts": 1787906195}
  ]
}

Semantics

null means not measured, 0 means measured zero. They are different answers and should be read differently.
start_ns is the base every cumulative stat accumulates from — the instance start locally, the caller’s start on the Graphsignal API.
gauge — the newest sampled value in the window (stats.value).
counter — the total accumulated over the window (stats.total).
histogram — the one distribution type. Exact count and sum accumulated over the window, min and max where the source keeps them (probes and the CUDA/ROCm libraries do; Prometheus does not), the derived mean, and p50/p95 computed from the bins (quantiles are bin values, so resolution is ≤25%). A source that reports totals without buckets — a Prometheus summary family — answers null for p50 and p95; that is a whole histogram, not a partial one.
profile — cumulative value per named frame with the number of samples behind each, as stats.frames sorted by value descending.
errors carries the most recent warnings and errors up to the snapshot instant.
Metrics not updated for 10 minutes disappear from the local report as dead workers age out.

CUDA graphs and the kernel profile

Which of the two GPU-time profiles holds a graph-replaying engine’s decode time depends on the tracing granularity, and cuda_graph_trace_mode tells you which one you are reading:

`cuda_graph_trace_mode`	`cuda_kernels_nanoseconds`	`cuda_graphs_nanoseconds`
`0` — graph (default)	eagerly launched kernels only; kernels replayed from a CUDA graph are absent	one frame per unique graph structure (a 16-hex signature), value = cumulative replay time, samples = replays
`1` — node (`--cuda-graph-trace node`)	every kernel by symbol, graph-replayed ones included	present but empty

So a cuda_kernels_nanoseconds that accounts for far less GPU time than the workload spends is the expected reading in graph mode, not a gap in the data — read cuda_graphs_nanoseconds for the rest, or rerun with --cuda-graph-trace node to get the same time broken down per kernel. On ROCm the distinction does not exist: rocprofiler-sdk reports every dispatch individually, so rocm_kernels_nanoseconds always holds them all and no mode gauge is published.

Units

Units live in the field names, on all three endpoints:

a query parameter with no suffix is epoch seconds (start, end);
a _ts response field is epoch seconds;
a _ns response field is epoch nanoseconds.

Reading it over time

Each read is the latest snapshot, so trends come from diffing cumulative values between reads — when and how often to read is the caller’s choice. On the Graphsignal API, pass the previous snapshot’s payload_ns / 1e9 as the next start and the windows tile exactly, with nothing counted twice.