Learn how to monitor machine learning models deployed as Python Flask services.
Flask is a popular choice when deploying ML models into production environment for serving behind an API endpoint. Just like other applications, models should be monitored to ensure they are available and performing as expected. ML models introduce additional requirement of monitoring input and output data. I discuss this in more details in another post Monitoring Machine Learning Models for Silent Failures.
In this post I'll show how to monitoring Flask-based model serving with Graphsignal.
Here is a simple Flask serving example with integrated Graphsignal Logger with a couple of additional lines of code.
You can get an API key by signing up for a Graphsignal account.
import time import json import pandas as pd import numpy as np import torch import torch.nn as nn from torch.utils.data import DataLoader from flask import Flask from flask import request # Import Graphsignal logger import graphsignal from fraud_model import FraudModel # Configure Graphsignal logger graphsignal.configure(api_key='api key here') sess = graphsignal.session(deployment_name='fraud_detector_prod') model = FraudModel().double().cpu() model.load_state_dict(torch.load('/path/to/model/fraud_model.pt')) app = Flask(__name__) @app.route('/predict_fraud', methods = ['POST']) def predict_fraud(): input_df = pd.DataFrame(data=request.get_json()) loss = nn.MSELoss() scores =  data_loader = DataLoader(input_df.to_numpy()) with torch.no_grad(): for input in data_loader: output = model(input) scores.append([loss(output, input).data.item()]) # Log predictions sess.log_prediction_batch(features=input_df, outputs=scores) return json.dumps(scores)
The logger takes care of windowing prediction data, computing statistics and sending to Graphsignal's cloud.
After deploying the application with integrated logger and firing some requests, within a few minutes prediction statistics will be available in the dashboards.
Feel free to contact us for a short demo.