Quick Start

Installation

Install the Python logger by running

pip install graphsignal

Or clone and install the GitHub repository.

git clone https://github.com/graphsignal/graphsignal.git
python setup.py install

Import the package in your application

import graphsignal

Configuration

Configure the logger by specifying your API key.

graphsignal.configure(api_key='my_api_key')

To get an API key, sign up for an account at graphsignal.com. The API key can then be found in your account's Settings / API Keys page.

Logging session

Get logging session for a deployed model identified by deployment name. Multiple sessions can be used in parallel in case of multi-model scrips or servers.

sess = graphsignal.session(deployment_name='model1_prod')

with statement can be used for offline/batch logging to make sure all remaining buffered data is automatically uploaded at the end.

with graphsignal.session(deployment_name='model1_prod') as sess:

Log any model metadata such as model version or deployment information.

sess.log_metadata('key1', 'val1')

Prediction Logging

Log single or batch model prediction/inference data to monitor data schema changes and drift. Computed data statistics are uploaded at certain intervals and on process exit.

Log single prediction.

sess.log_prediction(
  features={'feat1': 1.2, 'feat2': 'XX'},
  output=True)

Log prediction batch. Pass prediction data using list, dict, numpy.ndarray or pandas.DataFrame.

sess.log_prediction_batch(
  features=[[1.2, 70], [3.5, 40]], 
  outputs=[[0.5], [0.75]])

See logging API reference for full documentation.

Evaluation Logging

Log prediction and ground truth label to evaluate model performance. Because ground truth is usually available at a later point, evaluation logging is independent from prediction logging. Prediction logging is not required for model performance monitoring and visualization.

sess.log_evaluation(
  prediction=False,
  label=True)

See logging API reference for full documentation.

Model output type is inferred from label and prediction types. Model performance metrics such as accuracy, F1-score, MSE, etc. are computed based on the model output type.

To additionally visualize and monitor performance metrics for various data segments, a segments list can be provided.

sess.log_evaluation(
  prediction=False,
  label=True, 
  segments=['age_group_2', 'country_US'])

Log evaluation batch by passing predictions and labels using list or numpy.ndarray.

sess.log_evaluation_batch(
  predictions=[True, True, False], 
  labels=[False, True, False],
  segments=[['state_CA'], ['state_CA'], ['state_MA']])

Dashboards and Alerting

After logging is setup, sign in to Graphsignal to check out various dashboards and set up alerts for automatically detected issues.

Examples

Online prediction logging

from tensorflow import keras
import json
from flask import Flask, request

import graphsignal
graphsignal.configure(api_key='my_key')

sess = graphsignal.session(deployment_name='fraud_detection_prod')
sess.log_metadata('model version', '1.0')

model = keras.models.load_model('fraud_model.h5')
app = Flask(__name__)

@app.route('/predict_fraud', methods = ['POST'])
def predict_digit():
    features = request.get_json()

    # feature extraction code here

    output_data = model.predict([input_data])

    sess.log_prediction(
      features=features, 
      prediction=output_data[0])

    return json.dumps(output_data.tolist())

app.run(port=8090)

Offline evaluation logging

import graphsignal
graphsignal.configure(api_key='my_key')

# load predictions and labels here
last_hour_predictions=...
last_hour_labels=...

with graphsignal.session(deployment_name='my_risk_model_prod') as sess:
  sess.log_evaluation_batch(
    predictions=last_hour_predictions,
    labels=last_hour_labels)

Performance

Graphsignal logger uses streaming algorithms for computing data statistics to ensure production-level performance and memory usage. Data statistics are computed for time windows and sent to Graphsignal by the background thread.

Since only data statistics is sent to our servers, there is no limitation on logged data size.

Security and Privacy

See Security and Privacy section.

Troubleshooting

To enable debug logging, add debug_mode=True to configure(). If the debug log doesn't give you any hints on how to fix a problem, please report it to our support team via your account.

In case of connection issues, please make sure outgoing connections to https://log-api.graphsignal.com are allowed.