Skip to content
Graphsignal Docs

SGLang Profiling

See the Quick Start guide on how to install Graphsignal.

graphsignal-run recognizes sglang ... and python -m sglang.launch_server ... invocations and enables SGLang’s metrics plus GPU profiling for you. Everything measured is served at the local /signals endpoint while the server runs.

What’s captured

Section titled “What’s captured”
  • GPU profiling: cumulative time per CUDA kernel, per CUDA graph, and per memcpy/memset/synchronization kind, plus transfer byte counters. See CUDA Profiling for details on what’s captured GPU-side.
  • Engine metrics: SGLang’s /metrics endpoint is scraped automatically (sglang:* counters, gauges, histograms). Graphsignal adds --enable-metrics to the command — SGLang’s Prometheus endpoint is off by default. The scrape port derives from SGLang’s --port (default 30000); override with --metrics-port.
  • System metrics: CPU, host memory, and GPU telemetry via NVML — collected by the profiler sidecar regardless of engine.
  • Errors: warnings, errors, and tracebacks extracted from SGLang’s console output, including crashes.

Run SGLang with graphsignal-run

Section titled “Run SGLang with graphsignal-run”
Terminal window
graphsignal-run sglang serve \
  --model-path Qwen/Qwen2.5-1.5B-Instruct \
  --port 8000
# or
graphsignal-run python -m sglang.launch_server \
  --model-path Qwen/Qwen2.5-1.5B-Instruct \
  --port 8000

While the server runs, read everything at:

Terminal window
curl -s http://127.0.0.1:18259/signals

CUDA graphs

Section titled “CUDA graphs”

SGLang decode paths replay CUDA graphs. Kernels launched eagerly appear in the cuda_kernels_nanoseconds profile by symbol; graph replays appear in the cuda_graphs_nanoseconds profile as one frame per unique graph structure, with cumulative replay time. To rank the individual kernels inside those replays instead, relaunch with --cuda-graph-trace node: the decode kernels then appear in cuda_kernels_nanoseconds by symbol and cuda_graphs_nanoseconds stays empty. See CUDA Profiling.

Add Graphsignal to an SGLang Docker image

Section titled “Add Graphsignal to an SGLang Docker image”

If your image does not include Graphsignal (or CUPTI), install Graphsignal at container startup and run SGLang through graphsignal-run.

Terminal window
docker run --gpus all \
  -p 8000:8000 \
  --ipc=host \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --entrypoint bash \
  lmsysorg/sglang:latest \
  -lc 'pip install --no-cache-dir graphsignal[cu12] \
        && exec graphsignal-run python -m sglang.launch_server \
            --model-path Qwen/Qwen2.5-1.5B-Instruct \
            --port 8000'

To upload signals to Graphsignal, add -e GRAPHSIGNAL_API_KEY=... — see the production feedback loop.