ONNX Runtime Inference Profiling

See the Quick Start Guide on how to install and configure the profiler.

To profile ONNX Runtime, add the following code before inference session initialization and around inference. All inferences will be measured, but only a few will be profiled to ensure low overhead. See profiling API reference for full documentation.

Profile ONNX Runtime using with context manager:

import onnxruntime
from graphsignal.profilers.onnxruntime import initialize_profiler, profile_inference

sess_options = onnxruntime.SessionOptions()
initialize_profiler(sess_options)

session = onnxruntime.InferenceSession('my_model_path', sess_options)
with profile_inference(session):
    session.run(...)

Examples

The ONNX MNIST example illustrates how to add initialize_profiler and profile_inference methods.

Distributed workloads

Graphsignal provides built-in support for distributed inference. Depending on the platform, it may be necessary to provide a run ID to all workers. Refer to Distributed Workloads section for more information.