This section contains tutorials to help you get started with the Hydrosphere platform. A tutorial shows how to accomplish a goal rather than a single basic task.
Typically, a tutorial has several sections. When a tutorial section has several pieces of code to illustrate it, they can be shown as a group of tabs that you can switch between.
For guides on performing more basic technical steps, please look in the How-To section:
Estimated completion time: 14 min.
In this tutorial, you will learn how to retrospectively compare the behavior of two different models.
By the end of this tutorial you will know how to:
Set up an A/B application
Analyze production data
Install the dependencies in your local environment.
We train and upload our model with 5 components as movie_rec:v1
Next, we train and upload a new version of our original model with 20 components as movie_rec:v2
We can check that we have multiple versions of our model by running:
To create an A/B deployment we need to create an Application with a single execution stage consisting of two model variants. These model variants are our Model A and Model B correspondingly.
The following code will create such an application:
movie-ab-appWe'll simulate production data flow by repeatedly asking our model for recommendations.
Estimated completion time: 11m.
This tutorial is relevant only for Kubernetes installation of Hydrosphere. Please refer to How to Install Hydrosphere on Kubernetes cluster.
In this tutorial, you will learn how to configure deployed Applications.
By the end of this tutorial you will know how to:
Train and upload an example model version
Create a Deployment Configuration
Create an Application from the uploaded model version with previously created deployment configuration
Examine settings of a Kubernetes cluster
In this section, we describe the resources required to create and upload an example model used in further sections. If you have no prior experience with uploading models to the Hydrosphere platform we suggest that you visit the Getting Started Tutorial.
Here are the resources used to train sklearn.ensemble.GradientBoostingClassifier and upload it to the Hydrosphere cluster.
requirements.txt is a list of Python dependencies used during the process of building model image.
serving.yaml is a resource definition that describes how model should be built and uploaded to Hydrosphere platform.
train.py is used to generate a model.joblib which is loaded from func_main.py during model serving.
Run python train.py to generate model.joblib
func_main.py is a script which serves requests and produces responses.
Our folder structure should look like this:
Do not forget to run python train.py to generate model.joblib!
After we have made sure that all files are placed correctly, we can upload the model to the Hydrosphere platform by running hs apply from the command line.
Next, we are going to create and upload an instance of Deployment Configuration to the Hydrosphere platform.
Deployment Configurations describe with which Kubernetes settings Hydrosphere should deploy servables. You can specify Pod Affinity and Tolerations, the number of desired pods in deployment, ResourceRequirements, and Environment Variables for the model container, and HorizontalPodAutoScaler settings.
Created Deployment Configurations can be attached to Servables and Model Variants inside of Application.
Deployment Configurations are immutable and cannot be changed after they've been uploaded to the Hydrosphere platform.
You can create and upload Deployment Configuration to Hydrosphere via YAML Resource definition or via Python SDK.
For this tutorial, we'll create a deployment configuration with 2 initial pods per deployment, HPA, and FOO environment variable with value bar.
Create the deployment configuration resource definition:
To upload it to the Hydrosphere platform, run:
Create the application resource definition:
To upload it to the Hydrosphere platform, run:
You can check whether with_replicas was successful by calling kubectl get deployment -A -o wide and checking the READYcolumn.
To check whether with_hpa was successful you should get a list of all created Horizontal Pod Autoscaler Resources. You can do so by calling kubectl get hpa -A
The output is similar to:
To list all environment variables run kubectl exec my-model-1-tumbling-star -it /bin/bash and then execute the printenv command which prints ann system variables.
The output is similar to:
In this tutorial, you will learn how to train and deploy a model for a classification task based on the Adult Dataset. The whole process consists of such steps as preparation, model training, uploading a model to the cluster and making a prediction on test samples.
By the end of this tutorial you will know how to:
Prepare data
Train a model
Deploy a model with SDK
Explore models via UI
Deploy a model with CLI and resource definition
For this tutorial, you need to have Hydrosphere Platform deployed and Hydrosphere CLI (hs) along with Python SDK (hydrosdk) installed on your local machine. If you don't have them yet, please follow these guides first:
For this tutorial, you can use a local cluster. To ensure that, run hs cluster in your terminal. This command shows the name and server address of a cluster you’re currently using. If it shows that you're not using a local cluster, you can configure one with the following commands:
Let's start with downloading the dataset and moving it to some folder, e.g. it could be data/ folder. Next you need to setup your working environment using the following packages:
Model training always requires some amount of initial preparation, most of which is data preparation. Basically, the Adult Dataset consists of 14 descriptors, 5 of which are numerical and 9 categorical, including a class column. Categorical features are usually presented as strings. This is not an appropriate data type for sending it into a model, so we need to transform it first. Note that we apply a specific type (int64) for OrdinalEncoder to obtain integers for categorical descriptors after transformation. Transforming the class column usually is not necessary. Also we can remove rows that contain question marks in some samples. Once the preprocessing is complete, you can delete the DataFrame (df):
There are many classifiers that you can potentially use at this stage. In this example, we’ll apply Random Forest classifier. After preprocessing, the dataset will be separated into train and test subsets. The test set will be used to check whether our deployed model can process requests on the cluster. Training step usually requires iniating your model class and applying a specific training method, which is fit() method in our case. After the training step, we can save a model with joblib.dump() in a model/ model folder. Training data can be saved as a csv file, but don't forget to place index=False to ignore index column and avoid further confusions with reading it again.
The easiest way to upload a model to your cluster is by using Hydrosphere SDK. SDK allows Python developers to configure and manage the model lifecycle on the Hydrosphere platform. Before uploading a model, you need to connect to your cluster:
Next, we need to create an inference script to be uploaded to the Hydrosphere platform. This script will be executed each time you are instantiating a model servable. Let's name our function file func_main.py and store it in the src folder inside the directory where your model is stored. Your directory structure should look like this:
The code in the func_main.py should be as follows:
It’s important to make sure that variables will be in the right order after we transform our dictionary for a prediction. For that purpose in cols we preserve column names as a list sorted by order of their appearance in the DataFrame.
To start working with the model in a cluster, we need to install the necessary libraries used in func_main.py. You need to create requirements.txt in the folder with your model and add the following libraries to it:
After this, your model directory with all necessary dependencies should look as follows:
Now we are ready to upload our model to the cluster.
Hydrosphere Serving has a strictly typed inference engine, so before uploading our model we need to specify it’s signature with SignatureBuilder. A signature contains information about which method inside the func_main.py should be called, as well as shapes and types of its inputs and outputs. You can use X.dtypes to check what types of data you have for each column. You can use int64 fields for all our independent variables after transformation. Our class variable (income) initially consists of two classes with text names instead of numbers, which means that it should be defined as the string (str) in the signature. In addition, you can specify the type of profiling for each variable using ProfilingType so Hydrosphere could know what this variable is about and analyze it accordingly. For this purpose, we can create a dictionary, which could contain keys as our variables and values as our profiling types. Otherwise, you can describe them one by one as a parameter in the input. Finally, we can complete our signature with assigning our output variable by with_output method and giving it a name (e.g. y), type, shape and profiling type. Afterwards we can build our signature by the build() method.
Next, we need to specify which files will be uploaded to the cluster. We use path variable to define the root model folder and payload to point out paths to all files that we need to upload. At this point, we can combine all our efforts by using ModelVersionBuilderobject, which describes our models and other objects associated with models before the uploading step. It has different methods that are responsible for assigning and uploading different components. For example, we can:
Specify runtime environment for our model by with_runtime method
Assign priorly built signature by with_signature()
Upload model's elements by with_payload()
Lastly, upload traning data that was previously applied for your model's traininig process by with_trainig_data(). Please note that the training data is required if you want to utilize various services as Data Drift, Automatic Outlier Detection and Data Visualization.
Now we are ready to upload our model to the cluster. This process consists of several steps:
Once ModelVersionBuilderis prepared we can apply the upload method to upload it.
Then we can lock any interaction with the model until it will be successfully uploaded.
ModelVersion helps to check whether our model was successfully uploaded to the platform by looking for it.
To deploy a model you should create an Application - a linear pipeline of ModelVersions with monitoring and other benefits. For that purpose, we are able to apply ExecutionStageBuilder, which describes the model pipeline for an application. In turn, applications provide Predictor objects, which should be used for data inference purposes. Don't pay much attention to weight parameter, it is needed for A/B testing.
Predictors provide a predict method which we can use to send our data to the model. We can try to make predictions for our test set that has preliminarily been converted to a list of dictionaries. You can check the results using the name that we have used for an output of Signature and preserve it in any format you would prefer. Before making a prediction don't forget to make a small pause to finish all necessary loadings.
If you want to interact with your model via Hydrosphere UI, you can go to http://localhost. Here you can find all your models. Click on a model to view some basic information about it: versions, building logs, created applications, model's environments, and other services associated with deployed models.
You might notice that after some time there appears an additional model with the metric postscript at the end of the name. This is your automatically formed monitoring model for outlier detection. Learn more about the Automatic Outlier Detection feature here.
🎉 You have successfully finished this tutorial! 🎉
Next, you can:
Go to the next tutorial and learn how to create a custom Monitoring Metric and attach it to your deployed model.
Explore the extended part of this tutorial to learn how to use YAML resource definitions to upload a ModelVersion and create an Application.
Another way to upload your model is to apply a resource definition. This process repeats all the previous steps like data preparation and training. The difference is that instead of SDK, we are using CLI to apply a resource definition.
A resource definition is a file that defines the inputs and outputs of a model, a signature function, and some other metadata required for serving. Go to the root directory of the model and create a serving.yaml file. You should get the following file structure:
Model deployment with a resource definition repeats all the steps of that with SDK, but in one file. A considerable advantage of using a resource definition is that besides describing your model it allows creating an application by simply adding an object to the contract after the separation line at the bottom. Just name your application and provide the name and version of a model you want to tie to it.
To start uploading, run hs apply -f serving.yaml. To monitor your model you can use Hydrosphere UI as was previously shown.
Estimated Completion Time: 18m.
In this tutorial, you will learn how to create a custom anomaly detection metric for a specific use case.
Let's take a problem described in the previous Train & Deploy Census Income Classification Model tutorial as a use case and census income dataset as a data source. We will monitor a model that classifies whether the income of a given person exceeds $50.000 per year.
By the end of this tutorial you will know how to:
Train a monitoring model
Deploy of a monitoring model with SDK
Manage сustom metrics with UI
Upload a monitoring model with CLI
For this tutorial, you need to have Hydrosphere Platform deployed and Hydrosphere CLI (hs) along with Python SDK (hydrosdk) _**_installed on your local machine. If you don't have them yet, please follow these guides first:
This tutorial is a sequel to the previous tutorial. Please complete it first to have a prepared dataset and a trained model deployed to the cluster:
We start with the steps we used for the common model. First, let's create a directory structure for our monitoring model with an /src folder containing an inference scriptfunc_main.py. We also need training data used from the previous tutorial, which we can copy directly to just created directory:
As a monitoring metric, we will use IsolationForest. You can learn how it works here. In this example we are going to use PyOD library, which is dedicated specifically to anomaly detection algorithms. Let's install it first.
The whole process is similar to what we are usually doing with common machine learning models. Let's import necessary libraries, train our outlier detection model and save it in our working directory. Specifically for training we are supposed to use the same training data as for our prediction model.
This is what the pprobability distribution of our inliers looks like. It is directly dependent upon the method you choose. In our case we have applied a linear conversion, which transforms outlier scores by the range of [0, 1] using Min-Max values. Remember that the model must be fitted first. By choosing a contamination parameter we can adjust a threshold that will separate inliers from outliers accordingly. You have to be thorough in choosing it to avoid critical prediction mistakes. Otherwise, you can also stay with 'auto'. To create a monitoring metric, we have to deploy that Isolation Forest model as a separate model on the Hydrosphere platform.
First, let's create a new directory where we will store our inference script with declared serving function and its definitions. Put the following code inside the src/func_main.py file:
Next, we need to install the necessary libraries. Create a requirements.txt and add the following libraries to it:
Just like with common models, we can use SDK to upload our monitoring model and bind it to the trained one. The steps are almost the same, but with some slight differences:
First, since we want to predict the anomaly score instead of sample class, we need to change the type of output field from 'str' to 'float64'
Next we need to find our model on the cluster before assigning it to our prediction model. There is a specific method called .find() inside ModelVersion class
Finally, we need to apply a couple of new methods to create a metric. MetricSpec is responsible for creating a metric for a specific model, with specific MetricSpecConfig, which describe parameters of our metric like probability threshold and principle by which metric should detect outliers. In this case, .LESS denotes that every value below provided threshold is defined as an inlier.
Anomaly scores are obtained through traffic shadowing inside the Hydrosphere's engine after making a Servable, so you don't need to perform any additional manipulations.
Go to the UI to observe and manage all your models. Here you will find 3 models on the left panel:
adult_model - a model that we trained for prediction in the previous tutorial
adult_monitoring_model - our monitoring model
adult_model_metric - a model that was created by Automatic Outlier Detection
Click on the trained model and then on Monitoring. On the monitoring dasboard you now have two external metrics: the first one is auto_od_metric that was automatically generated by Automatic Outlier Detection, and the new one is custom_metric that we have just created. You can also change settings for existing metrics and configure the new ones in the Configure Metrics section:
During the prediction, you will get anomaly scores for each sample in the form of a chart with two lines. The curved line shows scores, while the horizontal dotted one is our threshold. When the curve intersects the threshold, it might be a sign of potential anomalousness. However, this is not always the case, since there are many factors that might affect this, so be careful about your final interpretation.
Just like in the case with all other types of models, we can define and upload a monitoring model using a resource definition. We have to pack our model with a model definition, like in the previous tutorial.
Inputs of this model are the inputs of the target monitored model plus the outputs of that model. We will use the value field as an output for the monitoring model. The final directory structure should look like this:
From that folder, upload the model to the cluster:
Now we have to attach the deployed Monitoring model as a custom metric. Let's create a monitoring metric for our pre-deployed classification model in the UI:
From the Models section, select the target model you would like to deploy and select the desired model version.
Open the Monitoring tab.
At the bottom of the page click the Configure Metric button.
From the opened window click the Add Metric button.
Specify the name of the metric.
Choose the monitoring model.
Choose the version of the monitoring model.
Select a comparison operator Greater. This means that if you have a metric value greater than a specified threshold, an alarm should be fired.
Set the threshold value. In this case, it should be equal to the value of monitoring_model.threshold_.
Click the Add Metric button.
That's it. Now you have a monitored income classifier deployed on the Hydrosphere platform.
Monitoring can be used to track the behavior of external models running outside of the Hydrosphere platform. This tutorial describes how to register an external model, trigger analysis over your requests, and retrieve results.
By the end of this tutorial you will know how to:
Register a model
Upload training data
Assign custom metrics
Invoke analysis
Retrieve metrics
For this tutorial, you need to have Hydrosphere Platform deployed on your local machine with Sonar component enabled. If you don't have it yet, please follow this guide first:
You also need a running external model, capable of producing predictions. Inputs and outputs of that model will be fed into Hydrosphere for monitoring purposes.
First, you have to register an external model. To do that, submit a JSON document, defining your model.
This section describes the structure of the JSON document used to register external models within the platform.
The document must contain the following top-level members, describing the interface of your model:
name: the name of the registered model. This name uniquely identifies a collection of model versions, registered within the Hydrosphere platform.
signature: the interface of the registered model. This member describes inputs and outputs of the model, as well as other complementary metadata, such as data profile for each field.
A document may contain additional top-level members, describing other details of your model.
metadata: the metadata of the registered model. The structure of the object is not strictly defined. The only constraint is that the object must have a key-value structure, where a value can only be of a simple data type (string, number, boolean).
monitoringConfiguration: monitoring configuration to be used for this model.
This example shows, how a model can be defined at the top level:
monitoringConfiguration object defines a monitoring configuration to be used for the model version. The object must contain the following members:
batchSize: size of the batch to be used for aggregations.
The example below shows how a monitoringConfiguration object can be defined.
signature object describes the signature of the model. The signature object must contain the following members:
signatureName: The signature of the model, used to process the request;
inputs: A collection of fields, defining the inputs of the model. Each item in the collection describes a single data entry, its type, shape, and profile. A collection must contain at least one item;
outputs: A collection of fields, defining the outputs of the model. Each item in the collection describes a single data entry, its type, shape, and profile. A collection must contain at least one item.
The example below shows how a predict object can be defined.
Items in the inputs / outputs collections are collectively called "fields". The field object must contain the following members:
name: Name of the field;
dtype: Data type of the field.
profile: Data profile of the field.
shape: Shape of the field.
The only valid options for dtype are:
DT_STRING;
DT_BOOL;
DT_VARIANT;
DT_HALF;
DT_FLOAT;
DT_DOUBLE;
DT_INT8;
DT_INT16;
DT_INT32;
DT_INT64;
DT_UINT8;
DT_UINT16;
DT_UINT32;
DT_UINT64;
DT_QINT8;
DT_QINT16;
DT_QINT32;
DT_QUINT8;
DT_QUINT16;
DT_COMPLEX64;
DT_COMPLEX128;
The only valid options for profile are:
NONE
NUMERICAL
TEXT
IMAGE
The example below shows how a single field object can be defined.
shape object defines the shape of the data that the model is processing. The shape object must contain the following members:
dims: An array of dimensions. A collection may be empty — in that case, the tensor will be interpreted as a scalar value.
The example below shows how a shape object can be defined.
A model can be registered by sending a POST request to the /api/v2/externalmodel endpoint. The request must include a model definition as primary data.
The request below shows an example of an external model registration.
As a response, the server will return a JSON object with complementary metadata, identifying a registered model version.
The response object from the external model registration request contains the following fields:
id: Model version ID, uniquely identifying a registered model version within Hydrosphere platform;
model: An object, representing a model collection, registered in Hydrosphere platform;
modelVersion: Model version number in the model collection;
signature: Contract of the model, similar to the one defined in the request section above;
metadata: Metadata of the model, similar to the one defined in the request section above;
monitoringConfiguration: MonitoringConfiguration of the model, similar to the one defined in the request section above;
created: Timestamp, indicating when the model was registered.
Note theid field. It will be referred as MODEL_VERSION_ID later throughout the article.
model object represents a collection of model versions, registered in the platform. The response model object contains the following fields:
id: ID of the model collection;
name: Name of the model collection.
The example below shows, a sample server response from an external model registration request.
To let Hydrosphere calculate the metrics of your requests, you have to submit the training data. You can do so by:
In each case your training data should be represented as a CSV document, containing fields named exactly as in the interface of your model.
Currently, we support uploading training data as .csv files and utilizing it for NUMERICAL, CATEGORICAL, and TEXT profiles only.
Switch to the cluster, suitable for your current flow.
If you don't have a defined cluster yet, create one using the following command.
Make sure you have a local copy of the training data that you want to submit.
Submit the training data. You must specify two parameters:
--model-version: A string indicating the model version to which you want to submit the data. The string should be formatted in the following way <model-name>:<model-version>;
--filename: Path to a filename, that you want to submit.
If you already have your training data uploaded to S3, you can specify a path to that object URI using --s3path parameter instead of --filename. The object behind this URI should be available to the Hydrosphere instance.
Depending on the size of your data, you will have to wait for the data to be uploaded. If you don't want to wait, you can use the --async flag.
To upload your data using an HTTP endpoint, stream it to the /monitoring/profiles/batch/<MODEL_VERSION_ID> endpoint.
You can acquire MODEL_VERSION_ID by sending a GET request to /model/version/<MODEL_NAME>/<MODEL_VERSION> endpoint. The response document will have a similar structure, already defined @refabove.
This step is optional. If you wish to assign a custom monitoring metric to a model, you can do it by:
using Hydrosphere UI
using HTTP endpoint
To find out how to assign metrics using Hydrosphere UI, refer to this page.
To assign metrics using HTTP endpoint, you will have to submit a JSON document, defining a monitoring specification.
The document must contain the following top-level members.
name: The name of the monitoring metric;
modelVersionId: Unique identifier of the model to which you want to assign a metric;
config: Object, representing a configuration of the metric, which will be applied to the model.
The example below shows how a metric can be defined on the top level.
config object defines a configuration of the monitoring metric that will monitor the model. The model must contain the following members:
modelVersionId: Unique identifier of the model that will monitor requests;
threshold: Threshold value, against which monitoring values will be compared using a comparison operator;
thresholdCmpOperator: Object, representing a comparison operator.
The example below shows, how a metric can be defined on a top-level.
thresholdCmpOperator object defines the kind of comparison operator that will be used when comparing a value produced by the metric against the threshold. The object must contain the following members:
kind: Kind of comparison operator.
The only valid options for kind are:
Eq;
NotEq;
Greater;
Less;
GreaterEq;
LessEq.
The example below shows, how a metric can be defined on the top level.
The request below shows an example of assigning a monitoring metric. At this moment, both monitoring and the actual prediction model should be registered/uploaded to the platform.
Monitoring service has GRPC API that you can use to send data for analysis. Here is how it works:
Need to use compiled GRPC services. We provide libraries with precompiled services. If your language is not there, you need to compile GRPC yourself.
Create an ExecutionMetadata message that contains information of the model that was used to process a given request.
Create a PredictRequest message that contains the original request passed to the model for prediction.
If model responded successfully then create a PredictResponse message that contains inferenced output of the model. If there was an error, then instead of a PredictResponse message you should prepare an error message.
Assemble an ExecutionInformation using the messages above.
Submit ExecutionInformation proto to Sonar for analysis. Use the RPC MonitoringService.Analyze method to calculate metrics.
Once triggered, the MonitoringService.Analyze method does not return anything. To fetch calculated metrics from the model version, you have to make a GET request to the /monitoring/checks/all/<MODEL_VERSION_ID> endpoint.
A request must contain the following parameters:
limit: how many requests to fetch;
offset: which offset to make from the beginning.
An example request is shown below.
Calculated metrics have a dynamic structure, which is dependant on the model interface.
A response object contains the original data submitted for prediction, the model's response, calculated metrics and other supplementary metadata. Every field produced by Hydrosphere is prefixed with _hs_ char.
_id: ID of the request, generated internally by Hydrosphere;
_hs_request_id: ID of the request, specified by user;
_hs_model_name: Name of the model that processed a request;
_hs_model_incremental_version: Version of the model that processed a request;
_hs_model_version_id: ID of the model version, which processed a request;
_hs_raw_checks: Raw checks calculated by Hydrosphere based on the training data;
_hs_metric_checks: Metrics produced by monitoring models;
_hs_latency: Latency, indicating how much it took to process a request;
_hs_error: Error message that occurred during request processing;
_hs_score: The number of all successful checks divided by the number of all checks;
_hs_overall_score: The amount of all successful metric values (not exceeding a specified threshold), divided by the amount of all metric values;
_hs_timestamp: Timestamp in nanoseconds, when the object was generated;
_hs_year: Year when the object was generated;
_hs_month: Month when the object was generated;
_hs_day: Day when the object was generated;
Apart from the fields defined above, each object will have additional fields specific to the particular model version and its interface.
_hs_<field_name>_score: The number of all successful checks calculated for this specific field divided by the total number of all checks calculated for this specific field;
<field_name>: The value of the field.
_hs_raw_checks object contains all fields, for which checks have been calculated.
The example below shows, how the _hs_raw_checks_ object can be defined.
check object declares the check, that has been calculated for the particular field. The following members will be present in the object.
check: Boolean value indicating, whether the check has been passed;
description: Description of the check that has been calculated;
threshold: Threshold of the check;
value: Value of the field;
metricSpecId: Metric specification ID. For each check object this value will be set to null.
The example below shows, how the check object can be defined.
_hs_metrics_checks object contains all fields for which metrics have been calculated.
The example below shows how the _hs_metrics_checks object can be defined.
metric object declares the metric, that has been calculated for the particular field. The following members will be present in the object.
check: Boolean value indicating, whether the metric has not been fired;
description: Name of the metric that has been calculated;
threshold: Threshold of the metric;
value: Value of the metric;
metricSpecId: Metric specification ID.
The example below shows how the metric object can be defined.
The example below shows a fully composed server response.
