Skip to content
Graphsignal Docs

ROCm Profiling

See the Quick Start guide on how to install Graphsignal.

Graphsignal profiles AMD ROCm workloads via AMD’s rocprofiler-sdk — no code instrumentation required. Attach the profiler to any HIP/ROCm program (PyTorch, vLLM, SGLang, raw HIP, custom Triton kernels) and the GPU side comes through automatically.

What’s captured

Section titled “What’s captured”

Everything lands in the local /signals endpoint as profile metrics — cumulative counters per named frame, sorted by value descending — plus counters:

  • rocm_kernels_nanoseconds — cumulative execution time per kernel, frames are the raw kernel symbols. Only with --rocm-kernel-trace all; empty by default (see below). The rocm_kernel_trace_mode gauge (0 none, 1 all) says which mode produced a payload.
  • rocm_memcpy_nanoseconds — cumulative time per transfer kind, with rocm_memcpy_bytes{kind} counters carrying the volumes.
  • rocm_sync_nanoseconds — cumulative host-blocking time per HIP synchronization API (hipDeviceSynchronize, hipStreamSynchronize, hipEventSynchronize, …).
  • gpu_* — device telemetry through AMD SMI (utilization, memory, power, temperature, clocks, PCIe, ECC), under the same names as the NVML telemetry on NVIDIA, with the device listed under resources.

Kernels replayed from a HIP graph are reported individually, exactly like eager launches — rocprofiler-sdk has no graph-level record, so --cuda-graph-trace is accepted and ignored on ROCm; --rocm-kernel-trace is the switch that matters here. The HIP runtime executes small copies (a few KB) as blit kernels; those show up in rocm_kernels_nanoseconds as __amd_rocclr_copyBuffer, while DMA transfers land in the memcpy instruments.

Host-side GPU probes work on ROCm too; device-storage probes recorded from HIP kernels are registered but not read by the ROCm activity library.

How it works

Section titled “How it works”

graphsignal-run loads a small activity library into the workload process via ROCP_TOOL_LIBRARIES; it collects rocprofiler-sdk activity records with low overhead and publishes cumulative state once per second. Everything else — analysis, statistics, the HTTP endpoint — runs in the sidecar profiler process, never inside the workload.

Integration

Section titled “Integration”

Wrap your launch command with graphsignal-run:

Terminal window
graphsignal-run <my-app>

Collection starts automatically once the workload begins using the GPU. Requires ROCm 7+; if ROCm isn’t found, Graphsignal skips ROCm profiling without affecting the workload. While the workload runs, read everything at http://127.0.0.1:18259/signals. See the Profiler CLI reference for options.

Kernel tracing is an investigation mode on ROCm

Section titled “Kernel tracing is an investigation mode on ROCm”