Tracing and Monitoring Local Models

Integration

Graphsignal tracer can be used in model applications that run models locally. See Quick Start guide on how to install and configure Graphsignal tracer.

Model deployments, versions or environments can be tracked separately. Simply set a different name to deployment in configure method.

import graphsignal

graphsignal.configure(api_key='my-api-key', deployment='my-app-prod')

with graphsignal.start_trace('predict', tags=dict(component='LLM')) as span:
    y = model(x)
    span.set_data('input', x, counts=dict(token_count=x_num_tokens))
    span.set_data('output', y)

or using decorator:

@trace_function(tags=dict(component='LLM'))
def generate(x):
    # run generation here

To record additional information with traced generations, provide tags to graphsignal.start_trace() method or use Trace.set_tag method.

For model servers that only accept model files and expose them via REST API, see Tracing And Monitoring Model APIs guide.