This section is to help users solve common known issues.
If you want to ask questions live, you are free to do so in our Hydrosphere channel on Slack:
Concepts
There are a few concepts that you should be familiar with before starting to work with the Hydrosphere platform.
Resource definitions
Resource definitions describe Models, Applications, and Deployment Configurations in the YAML format. You can learn more about them in the How to write resource definitions section.
Serving
Models & Model Versions
A Model is a machine learning model or a processing function that consumes provided inputs and produces predictions or transformations.
Within the Hydrosphere platform, we break down a model into its versions. Each Model version represents a single Docker image containing all the artifacts that you have uploaded to the platform. Consequently, Model is a group of Model versions with the same name.
Runtimes
A Runtime is a Docker image with the predefined gRPC interface which loads and serves your model.
We have a few runtimes, which you can use in your own projects.
Servable
Servable is a deployed instance of a Model version combined with a Runtime. It exposes a gRPC endpoint that can be used to send requests.
Users should not use Servables as-is, since they are designed to be building blocks, rather than inference endpoints. Hydrosphere provides a better alternative to deploy a Model version — Application.
Applications
An Application is a pipeline of one or more stages, each consisting of one or multiple Model Versions. Data sent to an application stage is shadowed to all of its model versions. The output of a stage is picked randomly with respect to weights.
When a user creates an Application, the Manager service automatically deploys appropriate Servables. The Application handles monitoring of your models and can perform A/B traffic splits.
Each Application has publicly available HTTP and gRPC endpoints that you can send requests to.
Deployment Configurations
A Deployment Configuration is a collection of Kubernetes settings that you can set for your Servables and Model Versions used inside of Application stages.
Deployment Configuration covers:
Horizontal Pod Autoscaler specs
Container Specs
Resource requirements: limits and requests
Model's Signature
A Model'sSignature is a specification of your model computation which identifies the name of a function with its inputs and outputs, including their names, shapes, and data types.
Example of a signature defined in a YAML file:
Field
A Field is a basic element of a Model's signature. It has a name, shape, data type, and profile.
Example of a model's signature field defined in a YAML file:
Field`s Profile
A Profile is a special tag that tells how Hydrosphere should interpret the field's data.
There are multiple available tags: Numerical, Categorical, Image, Text, Audio, Video, etc.
Monitoring
Metrics
Data coming through deployed Model Versions can be monitored with metrics.
Metric is a Model Version that takes a combination of inputs & outputs from another monitored Model Version, receives every request and response from the monitored model, produces a single value, and compares it with a threshold to determine whether this request was healthy or not.
Every request is evaluated against all metrics assigned to the model.
Auto OD Metric is an automatically generated Outlier Detection metric. More details are described .
Checks
A check is a boolean condition associated with a field of a Model Version signature which shows for every request whether the field value is acceptable or not.
For example, Min/Max checks ensure that a field value is in an acceptable range which is inferred from training data values.
Hydrosphere is a language-agnostic platform. You can use it with models written in any language and trained in any framework. Your ML models can come from any background, without restrictions of your choices regarding ML model development tools.
In Hydrosphere you operate ML models as Runtimes, which are Docker containers packed with predefined dependencies and gRPC interfaces for loading and serving them on the platform with a model inside. All models that you upload to Hydrosphere must have the corresponding runtimes.
The Hydrosphere component responsible for building Docker images from models for deployment, storing them in the registry, versioning, and more is Manager.
AWS Sagemaker
Traffic Shadowing
A/B Deployment
Hydrosphere users can use multiple model versions inside of the same Application stage. Hydrosphere shadows traffic to all model versions inside of an application stage.
Users can specify the likelihood that a model output will be selected as an application stage output by using the weight argument.
Traffic is shadowed to all versions, but only v2 and v3 return output
Traffic Shadowing
Hydrosphere shadows traffic to all model versions inside of an application stage.
If you want to shadow your traffic between model versions without producing output from them simply set weight parameter to 0. This way your model version will receive all incoming traffic, but its output will never be chosen as an output of an application stage.
Kubeflow Components
Hydrosphere Serving Components for Kubeflow Pipelines provide integration between Hydrosphere model serving benefits and orchestration capabilities. This allows launching training jobs as well as serving the same models in Kubernetes in a single pipeline.
You can find examples of sample pipelines .
Serving components
Serving
Gateway
Gateway is a service responsible for routing requests to/from or between Servables and Applications and validating these requests for matching a Model's/Application signature.
The Gateway maps a model’s name to a corresponding container. Whenever it receives a request via HTTP API, GRPC, or Kafka Streams, it communicates with that container via the gRPC protocol.
Platform Architecture
Hydrosphere is composed of several microservices, united to efficiently serve and monitor machine learning models in production. Hydrosphere features are divided between multiple services. You can learn more about each of them in this section.
UI / nginx
Alerts
Overview
**** sends data about any failed health checks of live production models and applications to Prometheus AlertManager. Once a user deploys a model to production, adds training data and starts sending production requests, these requests start getting checked by Sonar. If Sonar detects an anomaly (for example, a data check failed, or a metric value exceeded the threshold), AlertManager sends an appropriate alert.
Users can manage alerts by setting up AlertManager for Prometheus on Kubernetes. This can be helpful when you have models that you get too many alerts from and need to filter, group, or partly silence them. AlertManager can take care of grouping, inhibition, silencing of alerts, and routing them to the receiver integration of your choice. To configure alerts, modify the prometheus-am-configmap-<release_name>
Monitoring
Automatic Outlier Detection
Sonar
ConfigMap.
For more information about Prometheus AlertManager please refer to its official documentation.
Hydrosphere exposes libraries that can be used to interact with services. We use these libraries internally, but you can also use them to implement integrations.
Python
Protobuf compiled messages and services: https://pypi.org/project/hydro-serving-grpc/
Software Development Kit: https://pypi.org/project/hydrosdk/
Command Line Interface: https://pypi.org/project/hs/
Scala
Protobuf compiled messages and services: https://search.maven.org/search?q=a:serving-grpc-scala_2.13
Java
Protobuf compiled messages and services: https://search.maven.org/search?q=a:serving-grpc-java-11
Protobuf
API and entities: https://github.com/Hydrospheredata/hydro-serving-protos
Interpretability
Interpretability provides EDA (Exploratory Data Analysis) and explanations for predictions made by your models to make predictions understandable and actionable. It also produces explanations for monitoring metrics to let you know why a particular request was marked as an outlier. The component consists of 2 services:
Explanations
Data Projections
Both services are built with Celery to run asynchronous tasks from apps and consists of a client, a worker, and a broker that mediates in between. A client generates a task and initiates it by adding a message to a queue, а broker delivers it to a worker, then the worker executes the task.
Interpretability services use MongoDB as both a Celery broker and backend storage to save task results. To save and retrieve model training and production data, the Interpretability component uses S3 storage.
When Explanation or Data Projection receives a task they create a new temporary Servable specifically for the model they need to make an explanation for. They use this Servable to run data through it in order to make new predictions and delete it after.
Prediction Explanations
Prediction Explanations generate explanations of model predictions to help you understand them. Depending on the type of data your model uses, it provides an explanation as either a set of logical predicates if your data is in a tabular format or a saliency map if your data is in the image format. Saliency Map is a heat map that highlights parts of a picture that a prediction was based on.
Data Projections
Data Projection visualizes high-dimensional data in a 2D scatter plot with an automatically trained UMAP transformer to let you evaluate data structure and spot clusters, outliers, novel data, or any other patterns. It is especially helpful if your model works with high-dimensional data, such as images or text embeddings.
Model Registry
Hydrosphere has an internal Model Registry as centralized storage for Model Versions. When you build a Dockerized model and upload it to Hydrosphere or create new model versions, they get uploaded/stored to the configured model registry in the form of images. This organizes and simplifies model management across the platform and production lifecycle.
Deploy
The Deploy component allows you to upload a model, trained in a Kubeflow pipelines workflow to a Hydrosphere platform.
The Release component allows you to create an Application from a model previously uploaded to Hydrosphere platform. This application will be capable of serving prediction requests by HTTP or gRPC.
Building a Docker Image from your ML model for future deployment
Storing these images inside a Docker Registry deployed alongside with
manager service
Versioning these images as Model Versions
Creating running instances of these Model Versions called Servables
inside Kubernetes cluster
Combining multiple Model Versions into a linear graph with a single
endpoint called Application
Gateway enables data flow between different stages in an Application Pipeline
Sonar service is responsible for managing metrics, training and production data storage, calculating profiles, and shadowing data to the Model Versions which are used as an outlier detection metrics.
Drift Report
How-To
This section offers guides that address technical aspects of working with the Hydrosphere platform.
Hydrosphere allows you to A/B test your ML models in production.
A/B testing is a great way of measuring how well your models perform or which of your model versions is more effective and taking data-driven decisions upon this knowledge.
Production ML applications always have specific goals, for example driving as many users as possible to perform some action. To achieve these goals, it’s necessary to run online experiments and compare model versions using metrics in order to measure your progress against them. This approach allows to track whether your development efforts lead to desired outcomes.
To perform a basic A/B experiment on an application consisting of 2 variants of a model, you need to train and upload both versions to Hydrosphere, create an application with a single execution stage from them, invoke it by simulating production data flow, then analyze production data using metrics of your choice.
Learn how to set up an A/B application:
Overview
What is Hydrosphere?
Hydrosphere is an open-source MLOps platform for deploying, managing, and monitoring ML models in production with Kubernetes.
Hydrosphere supports all major machine learning frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn, fastai, etc. The platform is designed to effectively measure performance and health metrics of your production models, making it possible to spot early signs of performance drops and data drifts, get insights into why they happen.
Hydrosphere offers immediate value to ML-based products:
Сovers all aspects of the production ML lifecycle - model versioning & deployment, traffic & contract management, data monitoring, gaining insights.
Easy & fast management of production models that brings models to production in minutes by reducing time to upload, update, and roll your models into production.
Allows to create reproducible, observable, and explainable machine learning pipelines.
Why Hydrosphere?
Production ML is a dangerous place where numerous things can and usually do go wrong, making issues harder to discover and fix. Hydrosphere automates MLOps in the production part of the ML lifecycle combining best practices of CI/CD and DevOps and putting special emphasis on monitoring performance of ML models after their deployment.
MLOps problems Hydrosphere addresses:
Non-interpretable, biased models
Integration between the tools of each step of production ML lifecycle
Long time to find & debug issues with production ML Models
What Hydrosphere is not
Hydrosphere is not an ML model training framework. Before using Hydrosphere, you need to train your models with one of many existing frameworks for ML model training.
We suggest you use one of the orchestrators, such as Kubeflow or Airflow, to deliver your model to the Hydrosphere.
Hydrosphere Components
Hydrosphere platform includes all steps of a production ML model cycle - Versioning, Deployment, Monitoring, and Maintenance. This combination allows us to use a single tool to build an observable, reproducible, and scalable workflow, and start getting early warnings once anything goes wrong. These steps of an ML lifecycle are divided between three components that make up the Hydrosphere platform - Serving, Monitoring, and Interpretability.
Serving
Hydro Serving is responsible for framework-agnostic model deployment and management. It allows Data Scientists to upload, version, combine into linear pipelines and deploy ML models trained in any ML framework to a Docker/Kubernetes cluster and expose HTTP/gRPC/Kafka API to other parties.
Monitoring
Hydro Monitoring tracks model performance over time, raising alerts in case of detected issues. It provides a real-time updated UI, where you can monitor your models to see service health and usage. This constant monitoring of model health is crucial for any ML-based business as it’s tied to business and financial metrics.
Hydrosphere is capable of monitoring model quality with or without getting additional labeled data. Labeled data is often used in production drawing conclusions about the quality of a model’s predictions. Sometimes it is hard to get labeled data in production in a timely and cost-effective manner, especially when you deal with large volumes of complex data. Hydrosphere circumvents this issue by analyzing data that flows through a model as a proxy evaluating model quality to detect if ML models start to degrade and make unreliable predictions due to production data drifts from training data.
Interpretability
Hydrosphere Interpretability provides human-readable explanations of the predictions made by your ML models, as well as the explanations of monitoring analytics made by Hydrosphere Monitoring. It helps to evaluate and analyze models and understand what features influence their decisions. The Interpretability component demystifies your ML process, provides a new level of confidence about the reasons behind your models’ decisions and a certain level of trust business can rely on.
Monitoring Dashboard
Monitoring Dashboard lets you track your performance metrics and get a high-level view of your data health.
Monitoring Dashboard plots all requests streaming through a model version which are colored in respect with how "healthy" they are. On the horizontal axis we group our data by batches and on the vertical axis we group data by signature fields. In this plot cells are determined by their batch and field. Cells are colored from green to red, depending on the average request health inside the batch.
Monitoring Dashboard UI
Hydrosphere
Platform for deploying your Machine Learning to production
Hydrosphere is a platform for deploying, versioning, and monitoring your machine learning models in production. It is language-agnostic and framework-agnostic, with support for all major programming languages and frameworks - Python, Java, Tensorflow, Pytorch, etc.
What to do next?
⭐️ Star on Github
💦 Explore our tutorial
🥳 Join
Inference Pipelines
A Hydrosphere user can create a linear inference pipeline from multiple model versions. Such pipelines are called Applications.
Inference Pipeline with two stages
Reference
This section of the Hydrosphere documentation contains references.
Provides understanding and control of models’ performance in production via data and target metrics analysis.
Adds in-depth observability for your production models and data flowing through them.
Improves business metrics of ML-based products as a result of a reduction in MTTR and MTTD incidents related to ML models due to early alerts once data drifts happen.
Monitoring for Model Degradation and Performance Loss
Understanding the reasons behind wrong predictions
Prediction Explanation service is designed to help Hydrosphere users understand the underlying causes of changes in predictions coming from their models.
Tabular Explanation for class 0
Prediction Explanation generates explanations of predictions produced by your models and tells you why a model made a particular prediction. Depending on the type of data your model uses, Prediction Explanation provides an explanation as either a set of logical predicates (if your data is in a tabular format) or a saliency map (if your data is in the image format). A saliency map is a heat map that highlights parts of a picture that a prediction was based on.
Saliency map calculated by RISE.
Hydrosphere uses model-agnostic methods for explaining your model predictions. Such methods can be used on any machine learning model after they've been uploaded to the platform.
As of now, Hydrosphere supports explaining tabular and image data with Anchor and tools correspondingly.
CLI
Hydrosphere CLI, orhs, is a command-line interface designed to work with the Hydrosphere platform.
This command lets you operate cluster instances. A cluster points to your Hydrosphere instance. You can use this command to work with different Hydrosphere instances.
See hs cluster --help for more information.
hs apply
This command allows you to upload resources from YAML definitions to the cluster.
See hs apply --help for more information.
hs profile
This command lets you upload your training data to build profiles.
$ hs profile push - upload training data to compute its profiles.
$ hs profile status - show profiling status for a given model.
See hs profile --help for more information.
hs app
This command provides information about available applications.
$ hs app list - list all existing applications.
$ hs app rm - remove a certain application.
See hs app --help - for more information.
hs model
This command provides information about available models.
This section contains tutorials to help you get started with the Hydrosphere platform. A tutorial shows how to accomplish a goal rather than a single basic task.
Typically, a tutorial has several sections. When a tutorial section has several pieces of code to illustrate it, they can be shown as a group of tabs that you can switch between.
For guides on performing more basic technical steps, please look in the How-To section:
Runtimes
If you are using a framework for which a runtime is not yet implemented, you can open an issue in our Github.
Python
Code is available on .
Version
Image
Link
Use private pip repositories
To use private pip repository you must add customized pip.conf file pointing to your custom PyPI repository.
For example, your custom pip.conf file can look like this:
You can tell pip to use this pip.conffile in the install-command field inside serving.yaml:
Data Projection
Data Projection is a service that visualizes high-dimensional data in a 2D scatter plot with an automatically trained transformer to let you evaluate the data structure and spot clusters, outliers, novel data, or any other patterns. This is especially helpful if your model works with high-dimensional data, such as images or text embeddings.
Data Projection is an important tool, which helps to describe complex things in a simple way. One good visualization can show more than text or data. Monitoring and interpretation of machine learning models are hard tasks that require analyzing a lot of raw data: training data, production requests, as well as model outputs.
Essentially, this data is just numbers that in their original form of vectors and matrices do not have any meaning since it is hard to extract any meaning from thousands of vectors of numbers. In Hydrosphere we want to make monitoring easier and clearer that is why we created a data projection service that can visualize your data in a single plot.
Usage
To start working with Data Projection you need to create a model that has an output field with an embedding of your data. Embeddings are real-valued vectors that represent the input features in a lower dimensionality.
Create a model with an embedding field
Data Projection service delegates the creation of embeddings to the user. It expects that model will create embedding from input features and pass it as output vector. Thus embedding field is required, models without this field are not supported. Data Projection also expects that output labels field is called class and model confidence is called respectively confidence. Other outputs are ignored.
Inside Data Projection service you can see your requests features projected on a 2D space:
Each point in the plot presents a request. Requests with similar features are close to each other. You can select a specific request point and inspect what it consists of.
Above plot, there are several scores: global score, stability score, MSID score, etc. These scores reflect the quality of projection of multidimensional requests to 2D. To interpret scores you refer to technical documentation on Data Projection service.
In the Colorize menu, you can choose how to colorize model requests: by class, by monitoring metric or by confidence. Data Projection searchers specifically for output scalars class and confidence.
In the Accent Points menu, you can highlight the nearest in original space points to the selected one by picking the nearest variant. Counterfactuals will show you nearest points to selected but with a different predicted label.
Download the latest released_version release from the releases page:
Unpack the tar ball:
Set up an environment:
Install from source
Clone the serving repository:
Set up an environment:
To check the installation, open http://localhost/. By default, Hydrosphere UI is available at port 80.
Kubernetes installation
By default, Hydrosphere spins up a minimal installation applicable only for testing purposes. Consult this document for details about deploying production-ready Hydrosphere instance.
To install Hydrosphere on the Kubernetes cluster you should have the following prerequisites fulfilled.
PV support on the underlying infrastructure (if persistence is required)
Docker registry with pull/push access (if the built-in one is not used)
Install from charts repository
Add the Hydrosphere charts repository:
Install the chart from repo to the cluster:
Install from source
Clone the repository:
Build dependencies:
Install the chart:
After the chart has been installed, you have to expose the ui component outside of the cluster. For the sake of simplicity, we will just port-forward it locally.
Anomaly detection is focused on identifying data objects that are different from our expectations. It can be influenced by bad practices like noise, errors, or some unexpected events. Unusual data points can be also due to rare, but correct behaviour, which often results in interesting findings, motivating a further investigation. For these reasons, it is necessary to develop some techniques that could allow us to identify such unusual events. We assume that such events may induce some objects generated by a ”different mechanism”, which indicates that these objects might contain unexpected patterns that do not conform to a normal behaviour.
For each model with uploaded training data, Hydrosphere creates an outlier detection (Auto OD) metric, which assigns an outlier score to each request. A request is labeled as an outlier if the outlier score is greater than the 97th percentile of training data outlier scores distribution.
At first sight, anomaly detection is perceived as a classification problem that differentiate between normal and abnormal events. But that's usually not the case, since abnormalities are not presented enough to be a separate labeled class or even might be completely absent, which transfers the problem of outlier detection into the unsupervised context, a type of machine learning that looks for previously undetected patterns in a dataset with no pre-existing labels and with a minimum of human supervision. Within these constraints we have no choice but to rely on unsupervised machine learning algorithms. Practically those algorithms should look at the data and model normal behaviour as good as possible. After this step they can detect potentially risky events, without having a priori knowledge of what malicious and benign behavior looks like, by checking if a new event is dissimilar enough from the baseline.
But in order to make a right choice, it is essential to measure their performance in terms of specific metrics like accuracy, F1-score, ROC-AUC, etc. For this we would normally need labels that tell whether an event is in fact unusual. But, as we stated before, the data sets we are using do not have labels. How can we still estimate performance? For this purpose we can apply a metric, which is called Area Under Mass Volume curve, which was developd specifically for unsuprevised anomaly ranking. This kind of metric might be assumed as a performance metric similar to , but for unsupervised anomaly detection. Briefly speaking, MV measures the extent to which the spread of the distribution of anomaly score for training data differs from that of a randomly generated uniform distribution. You can learn more about this method as well as this metric by this .
Hydrosphere has a specific engine inside the platform that automatically creates an outlier detection metric and assigns it to each downloaded model accordingly. The whole process can be divided into several consecutive stages:
As a starting point, Hydrosphere utilizes a training data to check whether it has an appropriate format for each feature
Then it applies the Mass Volume curve method to find an appropriate anomaly detection model. At the moment Hydrosphere chooses among three anomaly detection algorithms: Isolation Forest, Local Outlier Factor, and One-Class Support Vector Machines with prewitten set of hyperparameters. Additional models will be added later.
Finally, it uploads the chosen model on the cluster and assigns it as an anomaly detection metric to the previously trained model
There is an important aspect of outlier detection algorithm, which is concerned about choosing an appropriate threshold. Most outlier detection models calculate outlier score for each sample of the training data and then establishes a threshold score for detecting potential anomalies. In order to find this value, there exist several thresholding techniques, which based on statistics like standard deviation around the mean, median absolute deviation and interquartile range. Unfortunately, these statistics can be significantly biased because of the presence of potential outliers like noise or errors, when calculating these measures. In Hydropshere, we did some preliminary experiments with different datasets to find a value that maximize predictive ability for anomaly detection models and established that 97th percentile of raw outlier scores looks most promising. It means that a request is labeled as an outlier if its anomaly score is greater than that of the 97th percentile of training data outlier scores distribution.
You can observe those assigned models deployed as metrics in your Monitoring dashboard. These metrics provide you with information about how novel/anomalous your data is. If these values of the metric deviate significantly from the common, you can tell that you experience some potential abnormality event. In the case, if you observe a gradually increasing number of such events, then it might be associated with a data drift, which makes a need to re-evaluate your ML pipeline to check for errors.
High-dimensional cases
As mentioned above, outlier detection has turned out to be an import problem in many research fields. Still for high-dimensional data detecting such rare behaviors is not a trivial task. High dimensionality refers to data sets that have a large number of independent variables, components, features, or attributes within the data available for analysis. The complexity of the data analysis increases with respect to the number of dimensions, requiring more sophisticated methods to process the data. As a result, different methods might suffer from diffrent problems. For example, in the high-dimensional perspective, distane between observations might be very small, which will reduce the efficiency of distance-based outlier detection methods. Or, for high-dimensional data some irrelevant attributes may impede the separability of outliers from normal samples. Despite that for some 'big data' cases Hydrosphere's models are able to detect a critical anomalousness, it is not recommended to entirely rely on the results of Auto OD for high-dimensional cases. You can choose a specific algorithm that are more adapted for such cases. Hydrosphere Automatic Outlier Detection allows you to train your own model that is not a part of the Hydrosphere's engine. There is a specific dedicated to a creation of you own custom outlier detection metric. As an example you can find couple of algorithms for this task, which are a part of the PyOD toolbox.
For more details about high-dimensional problem and algorithms dedicated to overcome this problem you can read .
Supported Models
Right now Auto OD feature works only for Models with numerical scalar fields and uploaded training data.
AWS infrastructure
Prerequisite
AWS account
EKS cluster 1.18 or greater
Registries
AWS has an Elastic Container Registries service, but this service can't auto-create repository on push. Therefore, we can't use it to store our serving docker image.
You can use an external registry, like a DockerHub, Artifactory, VMWare Harbour or your own registry. Also, you can use AWS marketplace to find registries solutions or just install a container registry with the certificate on EC2.
Set registry in helm
After uploading the model, hydro-serving-manager makes docker image, stores it in the registry (path https://urlregistry.example.com/modelname:modelversion) and kubelet service should be able to download it. In values.yaml or values-production.yaml set
Databases
We used 2 databases, PostgreSQL 10 or greater and MongoDB 4 or greater.
For AWS you can set up RDS PostgreSQL or AuroraDB with database engine postgresql:10 or greater
For MongoDB, you can use AWS marketplace or install it on EC2. DocumentDB doesn't work now.
Create postgres RDS instance
After creating the instance, connect to the RDS instance and create a new database with additional params like a maintenance window, backup options, etc.
Set RDS and MongoDB in helm
Persistence
We used s3 to store training data, models metrics, and docker registry storage.
You can use 1 bucket for all service. They will create a path or separate buckets for hydro-vizualisations and sonar.
We used minio as an s3 proxy. You can use any s3 like object storage. To use your own, specify url in the persistence block
If you want to use s3, set s3 and credentials:
In that case, minio will be installed with s3 backend mode and work as an s3 gateway.
All services try to create s3 buckets if they don't exist. By default:
sonar will create hydrosphere-feature-lake
vizualization will create hydrosphere-visualization-artifacts
docker-registry will create hydrosphere-model-registry
Tolerations
Some cases required a different environments. You can use different machine groups for different installations or just be sure, that hydrosphere installs only some type of nodes.
Tolerations help you set these rules.
This example configures all deployments to deploy only nodes with taint node: highPerformance
All our helm charts have resource params and java services have javaOpts. These params help to configure requests and limits resources. JavaOpts help tune JVM machine.
Resources set pod requests and limit params for CPU and memory.
registry:
insecure: false
url: "docker.io" # Example for dockerhub
username: example # Username to authenticate to the registry
password: example # Password to authenticate to the registry
mongodb:
url: "mongodb.example.com:27017" # Specify MongoDB connection string if you want to use an external MongoDB instance.
rootPassword: hydr0s3rving # Specify root password
username: root # Specify username
password: hydr0s3rving # Specify user password
authDatabase: admin # Specify authDatabase
retry: false # Specify retry options if it's required
database: hydro-serving-data-profiler # Specify mongoDB database
postgresql:
url: "hydrosphere-instance.us-east-1.amazonaws.com" # Specify Postgresql connection string if you want to use an external Postgresql instance.
username: root # Specify root username
password: hydr0s3rving # Specify root password
database: hydro-serving # Specify database name
persistence:
url: "" # Any s3 like object storage
accessKey: ACCESSKEYEXAMPLE # accesskeyid for s3 or minio
secretKey: SECRETKEYEXAMPLE # secretkeyid for s3 or minio
persistence:
mode: s3 # Defines the type of the persistence storage. Valid options are "s3" and "minio".
accessKey: ACCESSKEYEXAMPLE # accesskeyid for s3 or minio
secretKey: SECRETKEYEXAMPLE # secretkeyid for s3 or minio
region: us-east-1
Drift Report service creates a statistical report based on a comparison of training and production data distributions. It compares these two sets of data by a set of statistical tests and finds deviations.
Drift report uses multiple different tests with p=.95 for different features:
Numerical features:
Levene's test with a trimmed mean
Welch's t-test
Mood's test
Kolmogorov–Smirnov test
Categorical features:
Chi-Square test
Unseen categories
Supported Models
Right now Drift Report feature works only for Models with numerical scalar fields.
Invoke applications
Inferencing applications can be achieved using any of the methods described below.
Hydrosphere UI
To send a sample request using Hydrosphere UI, open the desired application, and press the Test button at the upper right corner. We will generate dummy inputs based on your model's contract and send an HTTP request to the model's endpoint.
To send an HTTP request, you should send a POST request to the /gateway/application/<applicationName> endpoint with the JSON body containing your request data, composed with respect to the model's contract.
Path Parameters
Name
Type
Description
Request Body
Name
Type
Description
gRPC
To send a gRPC request you need to create a specific client.
Python SDK
You can learn more about our Python SDK .
application:name
string
Name of the application
object
Request data, composed with respect to the model's contract.
import grpc import hydro_serving_grpc as hs # pip install hydro-serving-grpc# connect to your ML Lamba instancechannel = grpc.insecure_channel("<host>")stub = hs.PredictionServiceStub(channel)# 1. define a model, that you'll usemodel_spec = hs.ModelSpec(name="model")# 2. define tensor_shape for Tensor instancetensor_shape = hs.TensorShapeProto(dim=[hs.TensorShapeProto.Dim(size=-1), hs.TensorShapeProto.Dim(# 3. define tensor with needed datatensor = hs.TensorProto(dtype=hs.DT_DOUBLE,tensor_shape=tensor_shape,double_val=# 4. create PredictRequest instancerequest = hs.PredictRequest(model_spec=model_spec,inputs={"x": tensor})# call Predict methodresult = stub.Predict(request)
Develop runtimes
Sometimes our runtime images are not flexible enough. In that case, you might want to implement one yourself.
The key things you need to know to write your own runtime are:
How to implement a predefined gRPC service for a dedicated language
How to our contracts' protobufs work to describe entry points, such as inputs and outputs
How to create your own Docker image and publish it to an open registry
Generate GRPC code
There are different approaches to generating client and server gRPC code in . Let's have a look at how to do that in Python.
First, let's clone our repository and prepare a folder for the generated code:
To generate the gRPC code we need to install additional packages:
Our custom runtime will require contracts and tf protobuf messages. Let's generate them:
The structure of the runtime should now be as follows:
Implement Service
Now that we have everything set up, let's implement a runtime. Create a runtime.py file and put in the following code:
Let's quickly review what we have here. RuntimeManager simply manages our service, i.e. starts it, stops it, and holds all necessary data. RuntimeService is a service that actually implements thePredict(PredictRequest) RPC function.
The model will be stored inside the /model directory in the Docker container. The structure of /model is a follows:
Thecontract.protobin file will be created by the Manager service. It contains a binary representation of the message.
files directory contains all files of your model.
To run this service let's create an another file main.py.
Publish Runtime
Before we can use the runtime, we have to package it into a container.
To add requirements for installing dependencies, create a requirements.txt file and put inside:
Create a Dockerfile to build our image:
APP_PORT is an environment variable used by Hydrosphere. When Hydrosphere invokes Predict method, it does so via the defined port.
The structure of the runtime folder should now look like this:
Build and push the Docker image:
Remember that the registry has to be accessible to the Hydrosphere platform so it can pull the runtime whenever it has to run a model with this runtime.
That's it. You have just created a simple runtime that you can use in your own projects. It is an almost identical version of our . You can always look up details there.
import com.google.protobuf.Int64Value;
import io.grpc.ManagedChannel;
import io.grpc.ManagedChannelBuilder;
import io.hydrosphere.serving.tensorflow.DataType;
import io.hydrosphere.serving.tensorflow.TensorProto;
import io.hydrosphere.serving.tensorflow.TensorShapeProto;
import io.hydrosphere.serving.tensorflow.api.Model;
import io.hydrosphere.serving.tensorflow.api.Predict;
import io.hydrosphere.serving.tensorflow.api.PredictionServiceGrpc;
import java.util.Random;
public class HydrosphereClient {
private final String modelName; // Actual model name, registered within Hydrosphere platform
private final Int64Value modelVersion; // Model version of the registered model within Hydrosphere platform
private final ManagedChannel channel;
private final PredictionServiceGrpc.PredictionServiceBlockingStub blockingStub;
public HydrosphereClient2(String target, String modelName, long modelVersion) {
this(ManagedChannelBuilder.forTarget(target).build(), modelName, modelVersion);
}
HydrosphereClient2(ManagedChannel channel, String modelName, long modelVersion) {
this.channel = channel;
this.modelName = modelName;
this.modelVersion = Int64Value.newBuilder().setValue(modelVersion).build();
this.blockingStub = PredictionServiceGrpc.newBlockingStub(this.channel);
}
private Model.ModelSpec getModelSpec() {
/*
Helper method to generate ModelSpec.
*/
return Model.ModelSpec.newBuilder()
.setName(this.modelName)
.setVersion(this.modelVersion)
.build();
}
private TensorProto generateDoubleTensorProto() {
/*
Helper method generating random TensorProto object for double values.
*/
return TensorProto.newBuilder()
.addDoubleVal(new Random().nextDouble())
.setDtype(DataType.DT_DOUBLE)
.setTensorShape(TensorShapeProto.newBuilder().build()) // Empty TensorShape indicates scalar shape
.build();
}
public Predict.PredictRequest generatePredictRequest() {
/*
PredictRequest is used to define the data passed to the model for inference.
*/
return Predict.PredictRequest.newBuilder()
.putInputs("in", this.generateDoubleTensorProto())
.setModelSpec(this.getModelSpec())
.build();
}
public Predict.PredictResponse predict(Predict.PredictRequest request) {
/*
The actual use of RPC method Predict of the PredictionService to invoke prediction.
*/
return this.blockingStub.predict(request);
}
public static void main(String[] args) throws Exception {
HydrosphereClient client = new HydrosphereClient("<host>", "example", 2);
Predict.PredictRequest request = client.generatePredictRequest();
Predict.PredictResponse response = client.predict(request);
System.out.println(response);
}
}
import hydrosdk as hs
hs_cluster = hs.Cluster(http_address='{HTTP_CLUSTER_ADDRESS}',
grpc_address='{GRPC_CLUSTER_ADDRESS}',)
app = hs.Application.find(hs_cluster, "{APP_NAME}")
predictor = adult_servable.predictor()
data = ... # your data
predictor.predict(data)
For this tutorial, you need to have Hydrosphere Platform deployed and Hydrosphere CLI (hs) along with Python SDK (hydrosdk) _**_installed on your local machine. If you don't have them yet, please follow these guides first:
This tutorial is a sequel to the previous tutorial. Please complete it first to have a prepared dataset and a trained model deployed to the cluster:
Train a Monitoring Model
We start with the steps we used for the common model. First, let's create a directory structure for our monitoring model with an /src folder containing an inference scriptfunc_main.py. We also need training data used from the previous tutorial, which we can copy directly to just created directory:
As a monitoring metric, we will use IsolationForest. You can learn how it works . In this example we are going to use library, which is dedicated specifically to anomaly detection algorithms. Let's install it first.
The whole process is similar to what we are usually doing with common machine learning models. Let's import necessary libraries, train our outlier detection model and save it in our working directory. Specifically for training we are supposed to use the same training data as for our prediction model.
This is what the pprobability distribution of our inliers looks like. It is directly dependent upon the method you choose. In our case we have applied a linear conversion, which transforms outlier scores by the range of [0, 1] using Min-Max values. Remember that the model must be fitted first. By choosing a contamination parameter we can adjust a threshold that will separate inliers from outliers accordingly. You have to be thorough in choosing it to avoid critical prediction mistakes. Otherwise, you can also stay with 'auto'. To create a monitoring metric, we have to deploy that Isolation Forest model as a separate model on the Hydrosphere platform.
Deploy a Monitoring Model with SDK
First, let's create a new directory where we will store our inference script with declared serving function and its definitions. Put the following code inside the src/func_main.py file:
Next, we need to install the necessary libraries. Create a requirements.txt and add the following libraries to it:
Just like with common models, we can use SDK to upload our monitoring model and bind it to the trained one. The steps are almost the same, but with some slight differences:
First, since we want to predict the anomaly score instead of sample class, we need to change the type of output field from 'str' to 'float64'
Next we need to find our model on the cluster before assigning it to our prediction model. There is a specific method called .find() inside ModelVersion class
Anomaly scores are obtained through inside the Hydrosphere's engine after making a Servable, so you don't need to perform any additional manipulations.
Managing Custom Metrics with UI
Go to the UI to observe and manage all your models. Here you will find 3 models on the left panel:
adult_model - a model that we trained for prediction in the
adult_monitoring_model - our monitoring model
adult_model_metric
Click on the trained model and then on Monitoring. On the monitoring dasboard you now have two external metrics: the first one is auto_od_metric that was automatically generated by , and the new one is custom_metric that we have just created. You can also change settings for existing metrics and configure the new ones in the Configure Metrics section:
During the prediction, you will get anomaly scores for each sample in the form of a chart with two lines. The curved line shows scores, while the horizontal dotted one is our threshold. When the curve intersects the threshold, it might be a sign of potential anomalousness. However, this is not always the case, since there are many factors that might affect this, so be careful about your final interpretation.
Uploading a Monitoring model with CLI
Just like in the case with all other types of models, we can define and upload a monitoring model using a resource definition. We have to pack our model with a model definition, like in the previous tutorial.
Inputs of this model are the inputs of the target monitored model plus the outputs of that model. We will use the value field as an output for the monitoring model. The final directory structure should look like this:
From that folder, upload the model to the cluster:
Now we have to attach the deployed Monitoring model as a custom metric. Let's create a monitoring metric for our pre-deployed classification model in the UI:
From the Models section, select the target model you would like to deploy and select the desired model version.
Open the Monitoring tab.
At the bottom of the page click the Configure Metric button.
That's it. Now you have a monitored income classifier deployed on the Hydrosphere platform.
Using Deployment Configurations
Estimated completion time: 11m.
This tutorial is relevant only for Kubernetes installation of Hydrosphere. Please refer to .
Overview
Contributing Pull Requests
This guide is written for contributing to documentation. It doesn't contain any instructions on installing software prerequisites. If your intended contribution requires any software installations, please refer to their respective official documentation.
Prerequisites
Git installed on your local machine
GitHub account
Contents
PR Contribution Workflow
Basic Workflow Example
PR Acceptance policy
PR Contribution Workflow
Fork and clone this repository (git clone)
Create a feature branch against master (git checkout -b featurename)
Make changes in the feature branch
Commit your changes (git commit -am "Add a feature")
Push your changes to GitHub (git push origin feature)
Open a Pull Request and wait for your PR to get reviewed
Edit your PR to address the feedback (if any)
See your PR getting merged
1. Fork and Clone this Repository
In order to contribute, you need to make your own copy of the repository you're going to contribute to. You do this by forking the repository to your GitHub account and then cloning the fork to your local machine.
Fork this GitHub repository: on GitHub, navigate to the main page of the repository and click the Fork button in the upper-right area of the screen. This will create a fork (a copy of this repository in your GitHub account).
Clone the fork and switch to the project directory by running in your terminal:
2. Create a New Branch
It is important to make all your changes in a separate branch created off the master branch.
Before any modifications to the repository that you've just cloned, create a new branch off of the master branch.
Create a new branch off of the current one and switch to it:
To switch between branches, use the same command without the -b flag. For example, to switch back to the master branch:
This way you can switch between multiple branches when you work on multiple features at once.
Branch Naming Conventions
Give your branch a descriptive name so that others working on the project understand what you are working on. The branch name should include the name of the module that you're contributing to.
Name your branch according to the following template, replacing nginx with the name of the module you're contributing to:
3. Make Changes
Make changes you want to propose. Make sure you do this in a dedicated branch based on the master branch.
4. Commit Changes
Commit changes often to avoid accidental data loss. Make sure to provide your commits with descriptive comments.
Or add and commit all changed files with one command:
5. Push Changes to GitHub
Push your local changes to your fork on GitHub.
For example, if your remote repository is called origin and you want to push a branch named docs/fix:
6. Open a Pull Request
Navigate to your fork on GitHub. Press the "New pull request" button in the upper-left part of the page. Add a title and a comment. Once you press the "Create pull request" button, the maintainers of this repository will receive your PR.
7. Address Feedback
After you submit the PR, one or several of the Hydrosphere repository reviewers will provide you with actionable feedback. Edit your PR to address all of the comments. Reviewers do their best to provide feedback and approval in a timely fashion but note that response time may vary based on circumstances.
8. Your PR Gets Merged
Once your PR is approved by a reviewer, it gets accepted and merged with the main repository. Merged PRs will get included in the next Hydrosphere release.
Basic Workflow Example
PR Acceptance Policy
What will make your PR more likely to get accepted:
Having your fixes on a dedicated branch
Proper branch naming
Descriptive commit messages
PR title describing what changed
PR comment describing why/where it changed in <80 chars
Texts checked for spelling and typos (you can use Grammarly)
Code snippets checked with linters (when applicable)
PR Title and Comment Conventions
A PR title should describe what has changed. A PR comment should describe why and what/where. If your changes relate to a particular issue, a PR comment should contain an issue number. Please keep PR comments below 80 characters for readability.
PR title example:
PR comment example:
Minor edits (typos, spelling, formatting, adding small text pieces) may get waved through. More substantial changes normally require more time, reviewers, and back-and-forths, and you might get asked for a PR resubmission or dividing changes into more that one PR. Usually, PRs are getting merged right after the approval.
git clone https://github.com/Hydrospheredata/hydro-serving.git
cd hydro-serving
git checkout -b <your-branch-name>
git checkout master
feature/docs_nginx
git add .
git commit -m "Add description"
git commit -am "Add description"
git push <repo-name> <branch-name>
git push origin docs/fix
git clone https://github.com/Hydrospheredata/hydro-serving.git
cd hydro-serving
git checkout -b docs/fix
git status
git commit -am "Add description"
git push origin docs/fix
Updated docs: README.md and CONTRIBUTING.md.
Updated README.md (minor changes, fixed all typos).
Updated CONTRIBUTING.md (added a paragraph about
using linters, added sections:
"Use Linter to ensure correct syntax and formatting",
"Push your changes to GitHub" to close issue 42.
Issue #42
Upload a monitoring model with CLI
Finally, we need to apply a couple of new methods to create a metric. MetricSpec is responsible for creating a metric for a specific model, with specific MetricSpecConfig, which describe parameters of our metric like probability threshold and principle by which metric should detect outliers. In this case, .LESS denotes that every value below provided threshold is defined as an inlier.
- a model that was created by Automatic Outlier Detection
From the opened window click the Add Metric button.
Specify the name of the metric.
Choose the monitoring model.
Choose the version of the monitoring model.
Select a comparison operator Greater. This means that if you have a metric value greater than a specified threshold, an alarm should be fired.
Set the threshold value. In this case, it should be equal to the value of monitoring_model.threshold_.
In this section, we describe the resources required to create and upload an example model used in further sections. If you have no prior experience with uploading models to the Hydrosphere platform we suggest that you visit the Getting Started Tutorial.
Here are the resources used to train sklearn.ensemble.GradientBoostingClassifier and upload it to the Hydrosphere cluster.
requirements.txt is a list of Python dependencies used during the process of building model image.
serving.yaml is a resource definition that describes how model should be built and uploaded to Hydrosphere platform.
train.py is used to generate a model.joblib which is loaded from func_main.py during model serving.
Run python train.py to generate model.joblib
func_main.py is a script which serves requests and produces responses.
Our folder structure should look like this:
Do not forget to run python train.py to generate model.joblib!
After we have made sure that all files are placed correctly, we can upload the model to the Hydrosphere platform by running hs apply from the command line.
Create a Deployment Configuration
Next, we are going to create and upload an instance of Deployment Configuration to the Hydrosphere platform.
Deployment Configurations describe with which Kubernetes settings Hydrosphere should deploy servables. You can specify Pod Affinity and Tolerations, the number of desired pods in deployment, ResourceRequirements, and Environment Variables for the model container, and HorizontalPodAutoScaler settings.
Created Deployment Configurations can be attached to Servables and Model Variants inside of Application.
Deployment Configurations are immutable and cannot be changed after they've been uploaded to the Hydrosphere platform.
For this tutorial, we'll create a deployment configuration with 2 initial pods per deployment, HPA, and FOO environment variable with value bar.
Create the deployment configuration resource definition:
To upload it to the Hydrosphere platform, run:
Create an Application
Create the application resource definition:
To upload it to the Hydrosphere platform, run:
Examine Kubernetes Settings
Replicas
You can check whether with_replicas was successful by calling kubectl get deployment -A -o wide and checking the READYcolumn.
HPA
To check whether with_hpa was successful you should get a list of all created Horizontal Pod Autoscaler Resources. You can do so by calling kubectl get hpa -A
The output is similar to:
Environment Variables
To list all environment variables run kubectl exec my-model-1-tumbling-star -it /bin/bash and then execute the printenv command which prints ann system variables.
This is an entry-point tutorial to the Hydrosphere platform. Estimated completion time: 13 min.
Overview
In this tutorial, you will learn the basics of working with Hydrosphere. We will prepare an example model for serving, deploy it to Hydrosphere, turn it into an application, invoke it locally, and use monitoring. As an example model, we will take a simple logistic regression model fit with randomly generated data, with some noise added to it.
By the end of this tutorial you will know how to:
mkdir -p monitoring_model/src
!cp model/train_adult.csv monitoring_model/
cd monitoring_model
touch src/func_main.py
pip install pyod -U
import pandas as pd
from joblib import dump
import matplotlib.pyplot as plt
from pyod.models.iforest import IForest
X_train = pd.read_csv('train_adult.csv')
OD = IForest(contamination=0.03)
train_pred = OD.fit(X_train)
train_scores = OD.predict_proba(X_train)
plt.hist(
train_scores[:,1],
bins=30,
alpha=0.7,
density=True,
label="Test data outlier scores"
)
plt.vlines(1-OD.contamination, 0, 0.5, label = "Threshold")
plt.gcf().set_size_inches(8, 5)
plt.title('Outlier Scores')
plt.xlabel('Probability')
plt.ylabel('Frequency')
plt.legend()
joblib.dump(OD, "monitoring_model.joblib")
import numpy as np
from joblib import load
monitoring_model = load('/model/files/monitoring_model.joblib')
features = ['age', 'workclass', 'fnlwgt',
'education', 'educational-num', 'marital-status',
'occupation', 'relationship', 'race', 'gender',
'capital-gain', 'capital-loss', 'hours-per-week',
'native-country']
def predict(**kwargs):
x = np.array([kwargs[feature] for feature in features]).reshape(1, len(features))
predicted = monitoring_model.predict_proba(x)[:,1]
return {"value": predicted.item()}
import joblib
import pandas as pd
from sklearn.datasets import make_blobs
from sklearn.ensemble import GradientBoostingClassifier
# initialize data
X, y = make_blobs(n_samples=3000, n_features=30)
# create a model
model = GradientBoostingClassifier(n_estimators=200)
model.fit(X, y)
# Save training data and model
pd.DataFrame(X).to_csv("training_data.csv", index=False)
joblib.dump(model, "model.joblib")
func:main.py
import joblib
import numpy as np
# Load model once
model = joblib.load("/model/files/model.joblib")
def infer(x):
# Make a prediction
y = model.predict(x[np.newaxis])
# Return the scalar representation of y
return {"y": np.asscalar(y)}
Prepare a model for Hydrosphere
Serve a model on Hydrosphere
Create an Application
Invoke an Application
Use basic monitoring
Prerequisites
For this tutorial, you need to have Hydrosphere Platform deployed and Hydrosphere CLI (hs) along with Python SDK (hydrosdk) _**_installed on your local machine. If you don't have them yet, please follow these guides first:
To let hs know where the Hydrosphere platform runs, configure a new cluster entity:
Before you start
In the next two sections, we will prepare a model for deployment to Hydrosphere. It is important to stick to a specific folder structure during this process to let hs parse and upload the model correctly. Make sure that the structure of your local model directory looks like this by the end of the model preparation section:
train.py - a training script for our model
requirements.txt - provides dependencies for our model
model.joblib - a model artifact that we get as a result of model training
src/func_main.py - an inference script that defines a function for making model predictions
serving.yaml - a resource definition file to let Hydrosphere know which function to call from the func_main.py script and let the model manager understand model’s inputs and outputs.
Training a model
While Hydrosphere is a post-training platform, let's start with basic training steps to have a shared context.
As mentioned before, we will use the logistic regression model sklearn.LogisticRegression. For data generation, we will use the sklearn.datasets.make_regression (link) method.
First, create a directory for your model and create a new train.py inside:
Put the following code for your model in the train.py file:
Next, we need to install all the necessary libraries for our model. In your logistic_regression folder, create a requirements.txt file and provide dependencies inside:
Install all the dependencies to your local environment:
Train the model:
As soon as the script finishes, you will get the model saved to a model.joblib file.
Model preparation
Every model in the Hydrosphere cluster is deployed as an individual container. After a request is sent from the client application, it is passed to the appropriate Docker container with your model deployed on it. An important detail is that all model files are stored in the /model/files directory inside the container, so we will look there to load the model.
To run our model we will use a Python runtime that can execute any Python code you provide. Model preparation is pretty straightforward, but you have to create a specific folder structure described in the "Before you start" section.
Provide the inference script
Let's create the main file func_main.pyin the /src folder of your model directory:
Hydrosphere communicates with the model using TensorProto messages. If you want to perform a transformation or inference on the received TensorProto message, you will have to retrieve its contents, perform a transformation on it, and pack the result back to the TensorProto message. Pre-built python runtime automatically converts TensorProto messages to Numpy arrays, so the end-user doesn't need to interact with TensorProto messages.
To do inference you have to define a function that will be invoked every time Hydrosphere handles a request and passes it to the model. Inside that function, you have to call a predict (or similar) method of your model and return your predictions:
Inside func_main.py we initialize our model outside of the serving function infer.This process will not be triggered every time a new request comes in.
The infer function takes the actual request, unpacks it, makes a prediction, packs the answer, and returns it. There is no strict rule for naming this function, it just has to be a valid Python function name.
Provide a resource definition file
To let Hydrosphere know which function to call from the func_main.py file, we have to provide a resource definition file. This file will define a function to be called, inputs and outputs of a model, a signature function, and some other metadata required for serving.
Create a resource definition file serving.yamlin the root of your model directorylogistic_regression:
Inside serving.yaml we also providerequirements.txt andmodel.joblib as payload files to our model:
At this point make sure that the overall structure of your local model directory looks as shown in the "Before you start" section.
Although we have train.py inside the model directory, it will not be uploaded to the cluster since we are not listing it underpayload in the resource definition file.
Serving a Model
Now we are ready to upload our model to Hydrosphere. To do so, inside the logistic_regression model directory run:
If you cannot find your newly uploaded model and it is listed on your models' page, it is probably still in the building stage. Wait until the model changes its status to Released, then you can use it.
Creating an Application
Once you have opened your model in the UI, you can create an application for it. Basically, an application represents an endpoint to your model, so you can invoke it from anywhere. To learn more about advanced features, go to the Applications page.
Creating an Application from the uploaded model
Open http://localhost/applications and press the Add New Application button. In the opened window select the logistic_regression model, name your application logistic_regression and click the "Add Application" button.
Invoking an application
Invoking applications is available via different interfaces. For this tutorial, we will cover calling the created Application by gRPC via our Python SDK.
To install SDK run:
Define a gRPC client on your side and make a call from it:
Getting Started with Monitoring
Hydrosphere Platform has multiple tools for data drift monitoring:
Data Drift Report
Automatic Outlier Detection
Profiling
In this tutorial, we'll look at the monitoring dashboard and Automatic Outlier Detection feature.
Hydrosphere Monitoring relies heavily on training data. Users must provide training data to enable monitoring features.
Provide training data
To provide training data users need to add the training-data=<path_to_csv> field to the serving.yaml file. Run the following script to save training data used in previous steps as a training_data.csv file:
Next, add the training data field to the model definition inside the serving.yaml file:
Upload a model
Now we are ready to upload our model. Run the following command to create a new version of the logistic_regresion model:
Open the http://localhost/models page to see that there are now two versions of thelogistic_regression model.
For each model with uploaded training data, Hydrosphere creates an outlier detection metric, which assigns an outlier score to each request. This metric labels a request as an outlier if the outlier score is greater than the 97th percentile of training data outlier scores distribution.
Update an Application
Let's send some data to our new model version. To do so, we need to update our logistic_regression application. To update it, we can go to the Application tab and click the "Update" button:
Upgrading an application stage to a newer version
Send data to Application
After updating our Application, we can reuse our old code to send some data:
Monitor data quality
You can monitor your data quality in the Monitoring Dashboard:
The Monitoring dashboard plots all requests streaming through a model version as rectangles colored according to how "healthy" they are. On the horizontal axis, we group our data by batches and on the vertical axis, we group data by signature fields. In this plot, cells are determined by their batch and field. Cells are colored from green to red, depending on the average request health inside this batch.
Check data drift detection
To check whether our metric will be able to detect data drifts, let's simulate one and send data from another distribution. To do so, let's slightly modify our code:
You can validate that your model was able to detect data drifts on the monitoring dashboard.
Train & Deploy Census Income Classification Model
Overview
In this tutorial, you will learn how to train and deploy a model for a classification task based on the . The whole process consists of such steps as preparation, model training, uploading a model to the cluster and making a prediction on test samples.
By the end of this tutorial you will know how to:
hs cluster add --name local --server http://localhost
hs cluster use local
mkdir logistic_regression
cd logistic_regression
touch train.py
train.py
import joblib
from sklearn.datasets import make_blobs
from sklearn.linear_model import LogisticRegression
# initialize data
X, y = make_blobs(n_samples=300, n_features=2, centers=[[-5, 1],[5, -1]])
# create a model
model = LogisticRegression()
model.fit(X, y)
joblib.dump(model, "model.joblib")
import joblib
import numpy as np
# Load a model once
model = joblib.load("/model/files/model.joblib")
def infer(x1, x2):
# Make a prediction
y = model.predict([[x1, x2]])
# Return the scalar representation of y
return {"y": y.item()}
from sklearn.datasets import make_blobs
from hydrosdk import Cluster, Application
cluster = Cluster("http://localhost", grpc_address="localhost:9090")
app = Application.find(cluster, "logistic_regression")
predictor = app.predictor()
X, _ = make_blobs(n_samples=300, n_features=2, centers=[[-5, 1],[5, -1]])
for sample in X:
y = predictor.predict({"x1": sample[0], "x2": sample[1]})
print(y)
save:training:data.py
import pandas as pd
from sklearn.datasets import make_blobs
# Create training data
X, y = make_blobs(n_samples=300, n_features=2, centers=[[-5, 1],[5, -1]])
# Create pandas.DataFrame from it
df = pd.DataFrame(X, columns=['x1', 'x2'])
df['y'] = y
# Save it as .csv
df.to_csv("training_data.csv", index=False)
from sklearn.datasets import make_blobs
from hydrosdk import Cluster, Application
cluster = Cluster("http://localhost", grpc_address="localhost:9090")
app = Application.find(cluster, "logistic_regression")
predictor = app.predictor()
X, _ = make_blobs(n_samples=300, n_features=2, centers=[[-5, 1],[5, -1]])
for sample in X:
y = predictor.predict({"x1": sample[0], "x2": sample[1]})
print(y)
send:bad:data.py
from sklearn.datasets import make_blobs
from hydrosdk import Cluster, Application
cluster = Cluster("http://localhost", grpc_address="localhost:9090")
app = Application.find(cluster, "logistic_regression")
predictor = app.predictor()
# Change make_blobs arguments to simulate different distribution
X, _ = make_blobs(n_samples=300, n_features=2, centers=[[-10, 10],[0, 0]])
for sample in X:
y = predictor.predict({"x1": sample[0], "x2": sample[1]})
print(y)
Prepare data
Train a model
Deploy a model with SDK
Explore models via UI
Deploy a model with CLI and resource definition
Prerequisites
For this tutorial, you need to have Hydrosphere Platform deployed and Hydrosphere CLI (hs) along with Python SDK (hydrosdk) installed on your local machine. If you don't have them yet, please follow these guides first:
For this tutorial, you can use a local cluster. To ensure that, run hs cluster in your terminal. This command shows the name and server address of a cluster you’re currently using. If it shows that you're not using a local cluster, you can configure one with the following commands:
Data preparation
Let's start with downloading the dataset and moving it to some folder, e.g. it could be data/ folder. Next you need to setup your working environment using the following packages:
Model training always requires some amount of initial preparation, most of which is data preparation. Basically, the Adult Dataset consists of 14 descriptors, 5 of which are numerical and 9 categorical, including a class column. Categorical features are usually presented as strings. This is not an appropriate data type for sending it into a model, so we need to transform it first. Note that we apply a specific type (int64) for OrdinalEncoder to obtain integers for categorical descriptors after transformation. Transforming the class column usually is not necessary. Also we can remove rows that contain question marks in some samples. Once the preprocessing is complete, you can delete the DataFrame (df):
Training a model
There are many classifiers that you can potentially use at this stage. In this example, we’ll apply Random Forest classifier. After preprocessing, the dataset will be separated into train and test subsets. The test set will be used to check whether our deployed model can process requests on the cluster. Training step usually requires iniating your model class and applying a specific training method, which is fit() method in our case. After the training step, we can save a model with joblib.dump() in a model/ model folder. Training data can be saved as a csv file, but don't forget to place index=False to ignore index column and avoid further confusions with reading it again.
Deploy a model with SDK
The easiest way to upload a model to your cluster is by using Hydrosphere SDK. SDK allows Python developers to configure and manage the model lifecycle on the Hydrosphere platform. Before uploading a model, you need to connect to your cluster:
Next, we need to create an inference script to be uploaded to the Hydrosphere platform. This script will be executed each time you are instantiating a model servable. Let's name our function file func_main.py and store it in the src folder inside the directory where your model is stored. Your directory structure should look like this:
The code in the func_main.py should be as follows:
It’s important to make sure that variables will be in the right order after we transform our dictionary for a prediction. For that purpose in cols we preserve column names as a list sorted by order of their appearance in the DataFrame.
To start working with the model in a cluster, we need to install the necessary libraries used in func_main.py. You need to create requirements.txt in the folder with your model and add the following libraries to it:
After this, your model directory with all necessary dependencies should look as follows:
Now we are ready to upload our model to the cluster.
Hydrosphere Serving has a strictly typed inference engine, so before uploading our model we need to specify it’s signature with SignatureBuilder. A signature contains information about which method inside the func_main.py should be called, as well as shapes and types of its inputs and outputs. You can use X.dtypes to check what types of data you have for each column. You can use int64 fields for all our independent variables after transformation. Our class variable (income) initially consists of two classes with text names instead of numbers, which means that it should be defined as the string (str) in the signature. In addition, you can specify the type of profiling for each variable using ProfilingType so Hydrosphere could know what this variable is about and analyze it accordingly. For this purpose, we can create a dictionary, which could contain keys as our variables and values as our profiling types. Otherwise, you can describe them one by one as a parameter in the input. Finally, we can complete our signature with assigning our output variable by with_output method and giving it a name (e.g. y), type, shape and profiling type. Afterwards we can build our signature by the build() method.
Next, we need to specify which files will be uploaded to the cluster. We use path variable to define the root model folder and payload to point out paths to all files that we need to upload. At this point, we can combine all our efforts by using ModelVersionBuilderobject, which describes our models and other objects associated with models before the uploading step. It has different methods that are responsible for assigning and uploading different components. For example, we can:
Specify runtime environment for our model by with_runtime method
Assign priorly built signature by with_signature()
Upload model's elements by with_payload()
Lastly, upload traning data that was previously applied for your model's traininig process by with_trainig_data(). Please note that the training data is required if you want to utilize various services as Data Drift, Automatic Outlier Detection and Data Visualization.
Now we are ready to upload our model to the cluster. This process consists of several steps:
Once ModelVersionBuilderis prepared we can apply the upload method to upload it.
Then we can lock any interaction with the model until it will be successfully uploaded.
ModelVersion helps to check whether our model was successfully uploaded to the platform by looking for it.
To deploy a model you should create an Application - a linear pipeline of ModelVersions with monitoring and other benefits. For that purpose, we are able to apply ExecutionStageBuilder, which describes the model pipeline for an application. In turn, applications provide Predictor objects, which should be used for data inference purposes. Don't pay much attention to weight parameter, it is needed for A/B testing.
Predictors provide a predict method which we can use to send our data to the model. We can try to make predictions for our test set that has preliminarily been converted to a list of dictionaries. You can check the results using the name that we have used for an output of Signature and preserve it in any format you would prefer. Before making a prediction don't forget to make a small pause to finish all necessary loadings.
Explore the UI
If you want to interact with your model via Hydrosphere UI, you can go to http://localhost. Here you can find all your models. Click on a model to view some basic information about it: versions, building logs, created applications, model's environments, and other services associated with deployed models.
You might notice that after some time there appears an additional model with the metric postscript at the end of the name. This is your automatically formed monitoring model for outlier detection. Learn more about the Automatic Outlier Detection feature here.
🎉 You have successfully finished this tutorial! 🎉
Next Steps
Next, you can:
Go to the next tutorial and learn how to create a custom Monitoring Metric and attach it to your deployed model.
Explore the extended part of this tutorial to learn how to use YAML resource definitions to upload a ModelVersion and create an Application.
Deploy a model with CLI and Resource Definitions
Another way to upload your model is to apply a resource definition. This process repeats all the previous steps like data preparation and training. The difference is that instead of SDK, we are using CLI to apply a resource definition.
A resource definition is a file that defines the inputs and outputs of a model, a signature function, and some other metadata required for serving. Go to the root directory of the model and create a serving.yaml file. You should get the following file structure:
Model deployment with a resource definition repeats all the steps of that with SDK, but in one file. A considerable advantage of using a resource definition is that besides describing your model it allows creating an application by simply adding an object to the contract after the separation line at the bottom. Just name your application and provide the name and version of a model you want to tie to it.
To start uploading, run hs apply -f serving.yaml. To monitor your model you can use Hydrosphere UI as was previously shown.
from tqdm import tqdm
results = []
for x in tqdm(test_X.to_dict('records')):
result = predictor.predict(x)
results.append(result['y'])
print(results[:10])
A model definition must contain the following fields:
runtime: a string defining the runtime Docker image that will be used to run a model. You can learn more about runtimes here.
contract: an object defining the inputs and outputs of a model.
A model definition can contain the following fields:
payload: a list of files that should be added to the container.
install-command: a string defining a command that should be executed during the container build.
training-data: a string defining a path to the file that will be uploaded to Hydrosphere and used as a training data reference. It can be either a local file or a URI to an S3 object. At the moment we only support .csv files.
metadata: an object defining additional user metadata that will be displayed on the Hydrosphere UI.
The example below shows how a model can be defined on the top level.
Contract object
contract object must contain the following fields:
inputs: an object, defining all inputs of a model
outputs: an object, defining all outputs of a model
contract object can contain the following fields:
name: a string defining the signature of the model that should be used to process requests
Field object
field object must contain the following fields:
shape: either "scalar" or a list of integers, defining the shape of your data. If a shape is defined as a list of integers, it can have -1 value at the very beginning of the list, indicating that this field has an arbitrary number of "entities". -1 cannot be put anywhere aside from the beginning of the list.
type: a string defining the type of data.
field object can contain the following fields:
profile: a string, defining the profile type of your data.
The only valid options for type are:
bool — Boolean
string — String in bytes
half — 16-bit half-precision floating-point
float16 — 16-bit half-precision floating-point
float32 — 32-bit single-precision floating-point
double — 64-bit double-precision floating-point
float64 — 64-bit double-precision floating-point
uint8 — 8-bit unsigned integer
uint16 — 16-bit unsigned integer
uint32 — 32-bit unsigned integer
uint64 — 64-bit unsigned integer
int8 — 8-bit signed integer
int16 — 16-bit signed integer
int32 — 32-bit signed integer
int64 — 64-bit signed integer
qint8 — Quantized 8-bit signed integer
quint8 — Quantized 8-bit unsigned integer
qint16 — Quantized 16-bit signed integer
quint16 — Quantized 16-bit unsigned integer
complex64 — 64-bit single-precision complex
complex128 — 128-bit double-precision complex
The only valid options for profile are:
text — monitoring such fields will be done with text-oriented algorithms.
image — monitoring such fields will be done with image-oriented algorithms.
numerical — monitoring such fields will be done with numerical-oriented algorithms.
categorical — monitoring such fields will be done with categorical-oriented algorithms.
The example below shows how a contract can be defined on the top level.
Metadata object
metadata object can represent any arbitrary information specified by the user. The structure of the object is not strictly defined. The only constraint is that the object must have a key-value structure, where a value can only be of a simple data type (string, number, boolean).
The example below shows, how metadata can be defined.
The example below shows a complete definition of a sample model.
kind: Application
The application definition must contain one of the following fields:
singular: An object, defining a single-model application;
pipeline: A list of objects, defining an application as a pipeline of models.
Singular object
singular object represents an application consisting only of one model. The object must contain the following fields:
model: A string, defining a model version. It is expected to be in the form model-name:model-version.
The example below shows how a singular application can be defined.
Pipeline object
pipeline represents a list of stages, representing models.
stage object must contain the following fields:
model: A string defining a model version. It is expected to be in the form model-name:model-version.
stage object can contain the following fields:
weight: A number defining the weight of the model. All models' weights in a stage must add up to 100.
The example below shows how a pipeline application can be defined.
In this application, 100% of the traffic will be forwarded to the claims-preprocessing:1 model version and the output will be fed into claims-model. 80% of the traffic will go to the claims-model:1 model version, 20% of the traffic will go to the claims-model:2 model version.
kind: DeploymentConfiguration
The DeploymentConfiguration resource definition can contain the following fields:
minReplicas : minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down.
maxReplicas : integer, upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than minReplicas.
cpuUtilization : integer from 1 to 100, target average CPU utilization (represented as a percentage of requested CPU) over all the pods; if not specified the default autoscaling policy will be used.
Container object
The container object can contain:
resources : object with limits and requests fields. Closely resembles the k8s ResourceRequirements object
env : object with string keys and string values which is used to set environment variables.
Pod object
The hpa object is similar to the Kubernetes PodSpec object.
The pod object can contain
nodeSelector : selector which must be true for the pod to fit on a node. Selector which must match a node's labels for the pod to be scheduled on that node. More info.
affinity : pod's scheduling constraints. Represented by an Affinity object.
tolerations : array of .
Deployment object
The deployment object must contain:
replicaCount : integer, number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1.
Example
The example below shows how a deployment configuration can be defined.
In this tutorial, you will learn how to retrospectively compare the behavior of two different models.
By the end of this tutorial you will know how to:
Set up an A/B application
Analyze production data
Prerequisites
Set Up an A/B Application
Prepare a model for uploading
Install the dependencies in your local environment.
Upload Model A
We train and upload our model with 5 components as movie_rec:v1
Upload Model B
Next, we train and upload a new version of our original model with 20 components as movie_rec:v2
We can check that we have multiple versions of our model by running:
Create an Application
To create an A/B deployment we need to create an with a single execution stage consisting of two model variants. These model variants are our and correspondingly.
The following code will create such an application:
Invoking movie-ab-app
We'll simulate production data flow by repeatedly asking our model for recommendations.
import sys
import joblib
from lightfm import LightFM
from lightfm.datasets import fetch_movielens
if __name__ == "__main__":
no_components = int(sys.argv[1])
print(f"Number of components is set to {no_components}")
# Load the MovieLens 100k dataset. Only five
# star ratings are treated as positive.
data = fetch_movielens(min_rating=5.0)
# Instantiate and train the model
model = LightFM(no_components=no_components, loss='warp')
model.fit(data['train'], epochs=30, num_threads=2)
# Save the model
joblib.dump(model, "model.joblib")
src/func:main.py
import joblib
import numpy as np
from lightfm import LightFM
# Load model once
model: LightFM = joblib.load("/model/files/model.joblib")
# Get all item ids
item_ids = np.arange(0, 1682)
def get_top_rank_item(user_id):
# Calculate scores per item id
y = model.predict(user_ids=[user_id], item_ids=item_ids)
# Pick top 3
top_3 = y.argsort()[:-4:-1]
# Return {'top_1': ..., 'top_2': ..., 'top_3': ...}
return dict([(f"top_{i + 1}", item_id) for i, item_id in enumerate(top_3)])
Monitoring can be used to track the behavior of external models running outside of the Hydrosphere platform. This tutorial describes how to register an external model, trigger analysis over your requests, and retrieve results.
By the end of this tutorial you will know how to:
Register a model
Upload training data
Assign custom metrics
Invoke analysis
Retrieve metrics
Prerequisites
For this tutorial, you need to have Hydrosphere Platform deployed on your local machine with Sonar component enabled. If you don't have it yet, please follow this guide first:
You also need a running external model, capable of producing predictions. Inputs and outputs of that model will be fed into Hydrosphere for monitoring purposes.
Model registration
First, you have to register an external model. To do that, submit a JSON document, defining your model.
Request document structure
This section describes the structure of the JSON document used to register external models within the platform.
Top-level members
The document must contain the following top-level members, describing the interface of your model:
name: the name of the registered model. This name uniquely identifies a collection of model versions, registered within the Hydrosphere platform.
signature: the interface of the registered model. This member describes inputs and outputs of the model, as well as other complementary metadata, such as data profile for each field.
A document may contain additional top-level members, describing other details of your model.
metadata: the metadata of the registered model. The structure of the object is not strictly defined. The only constraint is that the object must have a key-value structure, where a value can only be of a simple data type (string, number, boolean).
monitoringConfiguration: monitoring configuration to be used for this model.
This example shows, how a model can be defined at the top level:
MonitoringConfiguration object
monitoringConfiguration object defines a monitoring configuration to be used for the model version. The object must contain the following members:
batchSize: size of the batch to be used for aggregations.
The example below shows how a monitoringConfiguration object can be defined.
Signature object
signature object describes the signature of the model. The signature object must contain the following members:
signatureName: The signature of the model, used to process the request;
inputs: A collection of fields, defining the inputs of the model. Each item in the collection describes a single data entry, its type, shape, and profile. A collection must contain at least one item;
The example below shows how a predict object can be defined.
Field object
Items in the inputs / outputs collections are collectively called "fields". The field object must contain the following members:
name: Name of the field;
dtype: Data type of the field.
profile
The only valid options for dtype are:
DT_STRING;
DT_BOOL;
DT_VARIANT;
The only valid options for profile are:
NONE
NUMERICAL
TEXT
The example below shows how a single field object can be defined.
Shape object
shape object defines the shape of the data that the model is processing. The shape object must contain the following members:
dims: An array of dimensions. A collection may be empty — in that case, the tensor will be interpreted as a scalar value.
The example below shows how a shape object can be defined.
Registering external model
A model can be registered by sending a POST request to the /api/v2/externalmodel endpoint. The request must include a model definition as primary data.
The request below shows an example of an external model registration.
As a response, the server will return a JSON object with complementary metadata, identifying a registered model version.
Response document structure
The response object from the external model registration request contains the following fields:
id: Model version ID, uniquely identifying a registered model version within Hydrosphere platform;
model: An object, representing a model collection, registered in Hydrosphere platform;
modelVersion
Note theid field. It will be referred as MODEL_VERSION_ID later throughout the article.
Model object
model object represents a collection of model versions, registered in the platform. The response model object contains the following fields:
id: ID of the model collection;
name: Name of the model collection.
The example below shows, a sample server response from an external model registration request.
Training data upload
To let Hydrosphere calculate the metrics of your requests, you have to submit the training data. You can do so by:
In each case your training data should be represented as a CSV document, containing fields named exactly as in the of your model.
Currently, we support uploading training data as .csv files and utilizing it for NUMERICAL, CATEGORICAL, and TEXT profiles only.
Upload using CLI
Switch to the cluster, suitable for your current flow.
If you don't have a defined cluster yet, create one using the following command.
Make sure you have a local copy of the training data that you want to submit.
Submit the training data. You must specify two parameters:
--model-version: A string indicating the model version to which you want to submit the data. The string should be formatted in the following way <model-name>:<model-version>;
--filename: Path to a filename, that you want to submit.
If you already have your training data uploaded to S3, you can specify a path to that object URI using --s3path parameter instead of --filename. The object behind this URI should be available to the Hydrosphere instance.
Depending on the size of your data, you will have to wait for the data to be uploaded. If you don't want to wait, you can use the --async flag.
Upload using an HTTP endpoint
To upload your data using an HTTP endpoint, stream it to the /monitoring/profiles/batch/<MODEL_VERSION_ID> endpoint.
You can acquire MODEL_VERSION_ID by sending a GET request to /model/version/<MODEL_NAME>/<MODEL_VERSION> endpoint. The response document will have a similar structure, already defined @ref.
Custom metrics assignment
This step is optional. If you wish to assign a custom monitoring metric to a model, you can do it by:
using Hydrosphere UI
using HTTP endpoint
Using Hydrosphere UI
To find out how to assign metrics using Hydrosphere UI, refer to page.
Using HTTP endpoint
To assign metrics using HTTP endpoint, you will have to submit a JSON document, defining a monitoring specification.
Top-level members
The document must contain the following top-level members.
name: The name of the monitoring metric;
modelVersionId: Unique identifier of the model to which you want to assign a metric;
config
The example below shows how a metric can be defined on the top level.
Config object
config object defines a configuration of the monitoring metric that will monitor the model. The model must contain the following members:
modelVersionId: Unique identifier of the model that will monitor requests;
threshold: Threshold value, against which monitoring values will be compared using a comparison operator;
The example below shows, how a metric can be defined on a top-level.
ThresholdCmpOperator object
thresholdCmpOperator object defines the kind of comparison operator that will be used when comparing a value produced by the metric against the threshold. The object must contain the following members:
kind: Kind of comparison operator.
The only valid options for kind are:
Eq;
NotEq;
Greater;
The example below shows, how a metric can be defined on the top level.
The request below shows an example of assigning a monitoring metric. At this moment, both monitoring and the actual prediction model should be registered/uploaded to the platform.
Analysis invocation
Monitoring service has GRPC API that you can use to send data for analysis. Here is how it works:
Need to use compiled GRPC services. We provide with precompiled services. If your language is not there, you need to compile GRPC yourself.
Create an message that contains information of the model that was used to process a given request.
Create a message that contains the original request passed to the model for prediction.
Metrics retrieval
Once triggered, the method does not return anything. To fetch calculated metrics from the model version, you have to make a GET request to the /monitoring/checks/all/<MODEL_VERSION_ID> endpoint.
A request must contain the following parameters:
limit: how many requests to fetch;
offset: which offset to make from the beginning.
An example request is shown below.
Calculated metrics have a dynamic structure, which is dependant on the model interface.
Response object structure
A response object contains the original data submitted for prediction, the model's response, calculated metrics and other supplementary metadata. Every field produced by Hydrosphere is prefixed with _hs_ char.
_id: ID of the request, generated internally by Hydrosphere;
_hs_request_id: ID of the request, specified by user;
_hs_model_name
Apart from the fields defined above, each object will have additional fields specific to the particular model version and its interface.
_hs_<field_name>_score: The number of all successful checks calculated for this specific field divided by the total number of all checks calculated for this specific field;
<field_name>: The value of the field.
Raw checks object
_hs_raw_checks object contains all fields, for which checks have been calculated.
The example below shows, how the _hs_raw_checks_ object can be defined.
Check object
check object declares the check, that has been calculated for the particular field. The following members will be present in the object.
check: Boolean value indicating, whether the check has been passed;
description: Description of the check that has been calculated;
threshold
The example below shows, how the check object can be defined.
Metrics object
_hs_metrics_checks object contains all fields for which metrics have been calculated.
The example below shows how the _hs_metrics_checks object can be defined.
Metric object
metric object declares the metric, that has been calculated for the particular field. The following members will be present in the object.
check: Boolean value indicating, whether the metric has not been fired;
description: Name of the metric that has been calculated;
threshold
The example below shows how the metric object can be defined.
The example below shows a fully composed server response.
Configuring Helm charts
This article explains the configuration file of the Hydrosphere Helm charts.
Prerequisistes
To install Hydrosphere on the Kubernetes cluster you should have the following prerequisites fulfilled.
outputs: A collection of fields, defining the outputs of the model. Each item in the collection describes a single data entry, its type, shape, and profile. A collection must contain at least one item.
: Data profile of the field.
shape: Shape of the field.
DT_HALF;
DT_FLOAT;
DT_DOUBLE;
DT_INT8;
DT_INT16;
DT_INT32;
DT_INT64;
DT_UINT8;
DT_UINT16;
DT_UINT32;
DT_UINT64;
DT_QINT8;
DT_QINT16;
DT_QINT32;
DT_QUINT8;
DT_QUINT16;
DT_COMPLEX64;
DT_COMPLEX128;
IMAGE
: Model version number in the model collection;
signature: Contract of the model, similar to the one defined in the request section above;
metadata: Metadata of the model, similar to the one defined in the request section above;
monitoringConfiguration: MonitoringConfiguration of the model, similar to the one defined in the request section above;
created: Timestamp, indicating when the model was registered.
: Object, representing a configuration of the metric,
which
will be applied to the model.
thresholdCmpOperator
: Object, representing a comparison operator.
Less;
GreaterEq;
LessEq.
If model responded successfully then create a PredictResponse message that contains inferenced output of the model. If there was an error, then instead of a PredictResponse message you should prepare an error message.
PV support on the underlying infrastructure (if persistence is required)
Docker registry with pull/push access (if the built-in one is not used)
Configuring Helm charts
Fetch the newest charts to your local directory.
Add the Hydrosphere charts repository:
Install the chart from repo to the cluster:
Helm charts are bundled with two distinct configuration files. The default one is values.yaml, the more comprehensive one is values-production.yaml.
By default (in the values.yaml), Helm charts are configured to set up a basic Hydrosphere installation aimed for a testing workload. To configure the installation for the production workload you have to set up additional resources, such as separate database instances, a separate docker registry, and override default values in the configuration file.
The contents of values.yaml and values-production.yaml files are overlapping, so we will continue with the latter.
Structure of values-production.yaml
Let's go over each section one by one.
UI
.global.ui.ingress.enabled is responsible for creating an ingress resource for the HTTP endpoint of the UI service.
.global.ui.ingress.host specifies the DNS name of the ingress resource.
.global.ui.ingress.path specifies the context path of the ingress resource.
.global.ui.ingress.enableGrpc is responsible for creating an ingress resource for the GRPC endpoint of the UI service. Note, specifying .global.ui.ingress.enableGrpc: true only works when the path is set to "/", so it's recommended to leave .global.ui.ingress.path untouched.
.global.ui.ingress.issuer is the name of the configured certificate issuer for ingress resources. Make sure it's set to either an Issuer or a ClusterIssuer. We do not bundle certificate manager to the Hydrosphere charts, so you have to set up this yourself. Consider consulting cert-manager.io documentation for more help.
.ui.resources section specifies resource requests and limits for the service.
Docker Registry
It is recommended to use a preconfigured docker registry for the production workload.
If you do not specify .global.registry.url,Hydrosphere will create an internal instance of the docker registry. This approach is only recommended for testing purposes.
.global.registry.url specifies the endpoint of your preconfigured docker registry.
.global.registry.username and .global.registry.password specify the credentials for your registry.
.global.registry.ingress.enabled is responsible for creating an ingress resource for the registry service. This also issues certificates for the docker registry, which are required for external registries.
If .global.registry.ingress.enabled is set to "true", .global.registry.insecure should be set to "false". This will tell Hydrosphere to work with the registry in secure mode.
If .global.registry.ingress.enabled is set to "false",.global.registry.insecure_should be set to "true"._ This will tell Hydrosphere to work with the registry in insecure mode. This will also create a DaemonSet which will proxy all requests to the registry from each node.
.global.registry.persistence section configures persistency options for the service. This is only valid when .global.persistence.mode is set to "s3".
.global.registry.persistence.bucket specifies the bucket name, where to store images.
.global.registry.persistence.region specifies region of the bucket. If not specified, it will be fallback to .global.persistence.region.
Persistence
It is recommended to use a preconfigured persistent storage for the production workload.
If you do not specify .global.persistence.url, Hydrosphere will create an internal instance of the minio storage. This approach is only recommended for testing purposes.
.global.persistence.url specifies the endpoint for your preconfigured storage.
.global.persistence.mode specifies, which persistence mode is used. Only valid options are "s3" or "minio".
.global.persistence.accessKey and .global.persistence.secretKey specify credentials to the storage.
.global.persistence.region specifies default regional constraint for the buckets.
Internal instance can be created when .global.persistence.mode is set to "minio".
MongoDB
It is recommended to use a preconfigured Mongo database instance for the production workload. .global.mongodb.url specifies the endpoint for your preconfigured Mongo instance.
If you omit specifying .global.mongodb.url, Hydrosphere will create an internal instance of the MongoDB database. This approach is only recommended for testing purposes.
Postgresql
It is recommended to use a preconfigured PostgreSQL database instance for the production workload. .global.postgresql.url specifies the endpoint for your preconfigured PostgreSQL instance.
If you omit specifying .global.postgresql.url, Hydrosphere will create an internal instance of the PostgreSQL database. This approach is only recommended for testing purposes.
AlertManager
.global.alertmanager.url specifies the endpoint for your preconfigured Prometheus AlertManager instance. If you omit specifying it, Hydrosphere will create an internal instance of AlertManager.
.global.alertmanager.config specifies configuration file for the AlertManager. Consider consulting AlertManager documentation for more details.
Manager
You can learn more about the Manager service in the Serving section.
.manager.javaOpts specifies Java options for the service.
.manager.serviceAccount section specifies ServiceAccount details for Manager service to use, when managing Kubernetes resources.
.manager.resources section specifies resource requests and limits for the service.
Gateway
You can learn more about the Gateway service in the Serving section.
.gateway.javaOpts specifies Java options for the service.
.gateway.resources section specifies resource requests and limits for the service.
Sonar
You can learn more about the Sonar service in the Monitoring section.
.sonar.javaOpts specifies Java options for the service.
.sonar.persistence section configures persistency options for the service.
.sonar.persistence.bucket specifies the bucket name, where to store training data and other artifacts.
.sonar.persistence.region specifies region of the bucket. If not specified, it will be fallback to .global.persistence.region.
.sonar.resources section specifies resource requests and limits for the service.
AutoOD
You can learn more about the AutoOd service in the Monitoring section.
.auto-od.resources section specifies resource requests and limits for the service.
Stat
You can learn more about the Stat service in the Monitoring section.
.stat.resources section specifies resource requests and limits for the service.
Visualization
You can learn more about the Visualization service in the Interpretability section.
.visualization.persistence section configures persistency options for the service.
.visualization.persistence.bucket specifies the bucket name, where to store data artifacts.
.visualization.persistence.region specifies region of the bucket. If not specified, it will be fallback to .global.persistence.region.
.visualization.resources section specifies resource requests and limits for the service.
RootCause
You can learn more about the RootCause service in the Interpretability section.
.rootcause.resources section specifies resource requests and limits for the service.
Tolerations
You can specify global tolerations for Hydrosphere services to be deployed on particular nodes using .global.tolerations. Consider consulting Kubernetes documentation for more details.
Installing charts
Once the charts were configured, install the release.
global:
ui:
ingress:
enabled: false
host: hydrosphere.local
path: "/"
enableGrpc: true # Enable ingress resources for grpc endpoints for services. Works only with `path: "/"`.
issuer: letsencrypt-prod
registry:
insecure: true
ingress: # optional, when url != ""
enabled: false
host: hydrosphere-registry.local
path: "/"
issuer: letsencrypt-prod
url: ""
username: example # Username to authenticate to the registry
password: example # Password to authenticate to the registry
persistence: # optional, when url != ""
bucket: hydrosphere-model-registry
region: us-east-1
persistence:
url: ""
mode: minio # Defines the type of the persistence storage. Valid options are "s3" and "minio".
accessKey: ACCESSKEYEXAMPLE # accesskeyid for s3 or minio
secretKey: SECRETKEYEXAMPLE # secretkeyid for s3 or minio
region: us-east-1 # optional, when mode == "minio"
mongodb:
url: "" # Specify MongoDB connection string if you want to use an external MongoDB instance.
# If empty, an in-cluster deployment will be provisioned.
rootPassword: hydr0s3rving
username: root
password: hydr0s3rving
authDatabase: admin
retry: false
database: hydro-serving-data-profiler
postgresql:
url: "" # Specify Postgresql connection string if you want to use an external Postgresql instance.
# If empty, an in-cluster deployment will be provisioned.
username: postgres
password: hydr0s3rving
database: hydro-serving
alertmanager:
url: "" # Prometheus AlertManager address in case you want to use the external installation.
# If empty, an internal installation will be deployed.
config:
global:
smtp_smarthost: localhost:25 # SMTP relay host
smtp_auth_username: mailbot # SMTP relay username
smtp_auth_identity: mailbot # SMTP relay username identity
smtp_auth_password: mailbot # SMTP relay password
smtp_from: [email protected] # Email address of the sender
route:
group_by: [alertname, modelVersionId]
group_wait: 10s
group_interval: 10s
repeat_interval: 1h
receiver: default
receivers:
- name: default
email_configs: # List of email addresses to send alarms to
- to: [email protected] tolerations: []
# - key: key
# operator: Equal
# value: value
# effect: NoSchedule
ui:
resources: {}
manager:
javaOpts: "-Xmx1024m -Xms128m -Xss16M"
servingAccount:
create: true
# name: "hydro-serving-manager-sa"
resources: {}
gateway:
javaOpts: "-Xmx512m -Xms64m -Xss16M"
resources: {}
sonar:
# A service, responsible for managing metrics, managing training and production data storage,
# calculating profiles, and shadowing data to the monitoring metrics.
javaOpts: "-Xmx2048m -Xmn2048m -Xss258k -XX:MaxMetaspaceSize=1024m -XX:+AggressiveHeap"
persistence:
bucket: "hydrosphere-feature-lake"
region: "us-east-1"
resources:
limits:
memory: 4Gi
requests:
memory: 512Mi
auto-od:
# A service, responsible for automatically generating outlier detection metrics for your
# production models based on the training data of the model.
resources: {}
stat:
# A service, responsible for creating statistical reports for your production models based
# on a comparison of training and production data distributions. Compares these two sets
# of data by a set of statistical tests and finds deviations.
resources: {}
vizualization:
# A service, responsible for visualizing high-dimensional data in a 2D scatter plot with
# an automatically trained transformer to let you evaluate the data structure and spot
# clusters, outliers, novel data, or any other patterns. This is especially helpful if
# your model works with high-dimensional data, such as images or text embeddings.
persistence:
bucket: hydrosphere-visualization-artifacts
region: us-east-1
resources: {}
rootcause:
# A service, responsible for generating explanations for a particular model prediction to
# help you understand the outcome by telling why your model made the prediction.
resources: {}
# Pull secret for hydrosphere from private registry
registry:
enabled: false
host: "" # Registry url for accessing hydrosphere images
username: "" # Registry username for accessing hydrosphere images
password: "" # Registry password for accessing hydrosphere images
