Monitoring External Models

Overview

Monitoring can be used to track the behavior of external models running outside of the Hydrosphere platform. This tutorial describes how to register an external model, trigger analysis over your requests, and retrieve results.

By the end of this tutorial you will know how to:

• Register a model

• Upload training data

• Assign custom metrics

• Invoke analysis

• Retrieve metrics

Prerequisites

For this tutorial, you need to have Hydrosphere Platform deployed on your local machine with Sonar component enabled. If you don't have it yet, please follow this guide first:

• Platform Installation

You also need a running external model, capable of producing predictions. Inputs and outputs of that model will be fed into Hydrosphere for monitoring purposes.

Model registration

First, you have to register an external model. To do that, submit a JSON document, defining your model.

Request document structure

This section describes the structure of the JSON document used to register external models within the platform.

Top-level members

The document must contain the following top-level members, describing the interface of your model:

• name: the name of the registered model. This name uniquely identifies a collection of model versions, registered within the Hydrosphere platform.

• signature: the interface of the registered model. This member describes inputs and outputs of the model, as well as other complementary metadata, such as data profile for each field.

A document may contain additional top-level members, describing other details of your model.

• metadata: the metadata of the registered model. The structure of the object is not strictly defined. The only constraint is that the object must have a key-value structure, where a value can only be of a simple data type (string, number, boolean).

• monitoringConfiguration: monitoring configuration to be used for this model.

This example shows, how a model can be defined at the top level:

{  
    "name": "external-model-example",
    "metadata": {
        "architecture": "Feed-forward neural network",
        "description": "Sample external model example",
        "author": "Hydrosphere.io",
        "training-data": "s3://bucket/external-model-example/data/",
        "endpoint": "http://example.com/api/external-model/"
    },
    "monitoringConfiguration": {
        "batchSize": 100
    },
    "signature": {
        ...
    }
}

MonitoringConfiguration object

monitoringConfiguration object defines a monitoring configuration to be used for the model version. The object must contain the following members:

• batchSize: size of the batch to be used for aggregations.

The example below shows how a monitoringConfiguration object can be defined.

{
    "batchSize": 100,
}

Signature object

signature object describes the signature of the model. The signature object must contain the following members:

• signatureName: The signature of the model, used to process the request;

• inputs: A collection of fields, defining the inputs of the model. Each item in the collection describes a single data entry, its type, shape, and profile. A collection must contain at least one item;

• outputs: A collection of fields, defining the outputs of the model. Each item in the collection describes a single data entry, its type, shape, and profile. A collection must contain at least one item.

The example below shows how a predict object can be defined.

{
    "signatureName": "predict",
    "inputs": [
        ...
    ],
    "outputs": [
        ...
    ]
}

Field object

Items in the inputs / outputs collections are collectively called "fields". The field object must contain the following members:

• name: Name of the field;

• dtype: Data type of the field.

• profile: Data profile of the field.

• shape: Shape of the field.

The only valid options for dtype are:

• DT_STRING;

• DT_BOOL;

• DT_VARIANT;

• DT_HALF;

• DT_FLOAT;

• DT_DOUBLE;

• DT_INT8;

• DT_INT16;

• DT_INT32;

• DT_INT64;

• DT_UINT8;

• DT_UINT16;

• DT_UINT32;

• DT_UINT64;

• DT_QINT8;

• DT_QINT16;

• DT_QINT32;

• DT_QUINT8;

• DT_QUINT16;

• DT_COMPLEX64;

• DT_COMPLEX128;

The only valid options for profile are:

• NONE

• NUMERICAL

• TEXT

• IMAGE

The example below shows how a single field object can be defined.

{
    "name": "age",
    "dtype": "DT_INT32",
    "profile": "NUMERICAL",
    "shape": {
        ...
    }
}

Shape object

shape object defines the shape of the data that the model is processing. The shape object must contain the following members:

• dims: An array of dimensions. A collection may be empty — in that case, the tensor will be interpreted as a scalar value.

The example below shows how a shape object can be defined.

"shape": {
    "dims": [-1, 64, 64],
}