Monitoring ML Models Deployed with Flask
By Dmitri Melikyan | | 1 min read

Learn how to monitor machine learning models deployed as Python Flask services.

Flask is a popular choice when deploying ML models into production environment for serving behind an API endpoint. Just like other applications, models should be monitored to ensure they are available and performing as expected. ML models introduce additional requirement of monitoring input and output data. I discuss this in more details in another post Monitoring Machine Learning Models for Silent Failures.

In this post I'll show how to monitoring Flask-based model serving with Graphsignal.

Example Flask application

Here is a simple Flask serving example with integrated Graphsignal Logger with a couple of additional lines of code.

You can get an API key by signing up for a Graphsignal account.

import time
import json
import pandas as pd
import numpy as np
import torch
import torch.nn as nn
from torch.utils.data import DataLoader
from flask import Flask
from flask import request

# Import Graphsignal logger
import graphsignal

from fraud_model import FraudModel

# Configure Graphsignal logger
graphsignal.configure(api_key='api key here')
sess = graphsignal.session(deployment_name='fraud_detector_prod')

model = FraudModel().double().cpu()
model.load_state_dict(torch.load('/path/to/model/fraud_model.pt'))

app = Flask(__name__)

@app.route('/predict_fraud', methods = ['POST'])
def predict_fraud():
    input_df = pd.DataFrame(data=request.get_json())
    
    loss = nn.MSELoss()
    scores = []
    data_loader = DataLoader(input_df.to_numpy())
    with torch.no_grad():
        for input in data_loader:
            output = model(input)
            scores.append([loss(output, input).data.item()])

    # Log predictions
    sess.log_prediction_batch(features=input_df, outputs=scores)

    return json.dumps(scores)

The logger takes care of windowing prediction data, computing statistics and sending to Graphsignal's cloud.

Analyzing data statistics and events

After deploying the application with integrated logger and firing some requests, within a few minutes prediction statistics will be available in the dashboards.

Feel free to contact us for a short demo.

data-analysis-dashboard