Skip to content
Graphsignal Docs

PyTorch Profiling

See the Quick Start guide on how to install Graphsignal.

Graphsignal profiles PyTorch workloads at the CUDA kernel level via CUPTI activity records — see CUDA Profiling for the full list of what’s captured (kernel profiles, CUDA graphs, memcpy/memset/sync, and NVML metrics). There is no PyTorch-specific instrumentation: PyTorch is a CUDA application like any other, and the profiler treats it as such.

What you’ll see

Section titled “What you’ll see”

PyTorch workloads predominantly emit kernels from cuBLAS, cuDNN, Triton, Flash Attention, and NCCL libraries. The cuda_kernels_nanoseconds profile reports cumulative time per raw GPU kernel symbol (e.g. sm80_xmma_gemm_f16f16_*, flash_attn_fwd_kernel, ncclAllReduceRingLLKernel_*), sorted by time descending, so the dominant kernels are at the top. Compiled or captured regions replayed as CUDA graphs appear in the cuda_graphs_nanoseconds profile instead, one frame per unique graph structure.

To time your own regions — steps, layers, custom ops — add GPU probes; their values appear next to the built-in metrics.

Integration

Section titled “Integration”

Wrap your launch command with graphsignal-run:

Terminal window
graphsignal-run python my_app.py

While the workload runs, read everything at:

Terminal window
curl -s http://127.0.0.1:18259/signals

To upload signals to Graphsignal, set GRAPHSIGNAL_API_KEY before launching — see the production feedback loop.