Skip to content
Graphsignal Docs

Profiler API

Graphsignal observes your workload from a sidecar process — the profiler. It runs out-of-process, never inside your workload. There are two ways to start it: the graphsignal-run CLI (see the Quick Start) and the graphsignal.watch() function documented below.

watch

Section titled “watch”
watch(otel_collector_port: Optional[int] = None,
      metrics_port: Optional[int] = None,
      cuda_graph_trace: Optional[str] = None) -> Optional[subprocess.Popen]

Spawn the profiler sidecar to observe the current process.

watch() does two things:

  1. Enables CUDA kernel profiling via CUPTI in the current process, which begins once CUDA initializes.
  2. Starts the profiler sidecar to observe the current process.

It must be called before any CUDA work happens in the current process. The returned Popen object lets the caller wait() or terminate() the sidecar.

import graphsignal


graphsignal.watch()
# ... your application code (PyTorch, vLLM, SGLang, etc.) ...

Arguments:

  • otel_collector_port: Optional TCP port for a local OTLP/gRPC collector that the sidecar will start. If your workload exports OTEL traces, point them at localhost:<otel_collector_port> so the profiler receives them. If omitted, no collector is started.
  • metrics_port: Optional TCP port for the sidecar to scrape Prometheus metrics from http://127.0.0.1:<metrics_port>/metrics. Use this when your workload exposes a /metrics endpoint. If omitted, no Prometheus scraping is performed.
  • cuda_graph_trace: Optional CUDA graph tracing mode: 'graph' (default) or 'node'. graph: one aggregated cuda.graph event per graph launch; lower CUPTI overhead. node: individual graph node activities as separate cuda.kernel / cuda.memcpy events; higher CUPTI overhead. Use 'node' when you need node-level visibility inside captured CUDA graphs. If omitted, graph-level tracing is used.

Returns:

  • subprocess.Popen for the spawned sidecar process, or None if the spawn failed.

Environment variables

Section titled “Environment variables”

The profiler subprocess reads its configuration from the environment. Set these before calling watch() (or before invoking graphsignal-run).

VariablePurpose
GRAPHSIGNAL_API_KEY (required)Your account API key.
GRAPHSIGNAL_API_BASEOverride the API endpoint (defaults to https://api.graphsignal.com).
GRAPHSIGNAL_TAG_<KEY>=<value>Arbitrary tag attached to all signals (e.g. GRAPHSIGNAL_TAG_DEPLOYMENT=us-prod).

To get an API key, sign up for a free account at graphsignal.com; the key is in Settings / API Keys.