ML Pipeline Monitoring and Profiling

Integration

Graphsignal can be used to monitor ML and data pipelines. Simply integrate the agent into pipeline stages separately. See Quick Start guide on how to install and configure the Graphsignal agent.

Give a unique endpoint to each pipeline stage. To further identify runs or jobs, provide tags to start_trace method.

for sample in dataset:
    with graphsignal.start_trace(endpoint='preprocessor-1', tags=dict(run_id=current_run_id)) as trace:
        trace.set_data('input', my_input_data)
        # data transformation
        trace.set_data('output', my_output_data)

for sample in dataset:
    with graphsignal.start_trace(endpoint='predictor-1', tags=dict(run_id=current_run_id)) as trace:
        trace.set_data('x', my_model_data)
        # model prediction