Hydrosphere has an internal Model Registry as centralized storage for Model Versions. When you build a Dockerized model and upload it to Hydrosphere or create new model versions, they get uploaded/stored to the configured model registry in the form of images. This organizes and simplifies model management across the platform and production lifecycle.

Gateway

Gateway is a service responsible for routing requests to/from or between Servables and Applications and validating these requests for matching a Model's/Application signature.

The Gateway maps a model’s name to a corresponding container. Whenever it receives a request via HTTP API, GRPC, or Kafka Streams, it communicates with that container via the gRPC protocol.

Manager

Manager is responsible for:

• Building a Docker Image from your ML model for future deployment

• Storing these images inside a Docker Registry deployed alongside with

  manager service

• Versioning these images as Model Versions

• Creating running instances of these Model Versions called Servables

  inside Kubernetes cluster

• Combining multiple Model Versions into a linear graph with a single

  endpoint called Application

Hydrosphere is a platform for deploying, versioning, and monitoring your machine learning models in production. It is language-agnostic and framework-agnostic, with support for all major programming languages and frameworks - Python, Java, Tensorflow, Pytorch, etc.

What to do next?

⭐️ Star Hydrosphere repo on Github

💦 Explore our Getting Started tutorial

🥳 Join Hydrosphere Slack Community

Automatic Outlier Detection

Sonar

Sonar service is responsible for managing metrics, training and production data storage, calculating profiles, and shadowing data to the Model Versions which are used as an outlier detection metrics.

Drift Report

Resource definitions

Resource definitions describe Models, Applications, and Deployment Configurations in the YAML format. You can learn more about them in the How to write resource definitions section.

Serving

Models & Model Versions

A Model is a machine learning model or a processing function that consumes provided inputs and produces predictions or transformations.

Within the Hydrosphere platform, we break down a model into its versions. Each Model version represents a single Docker image containing all the artifacts that you have uploaded to the platform. Consequently, Model is a group of Model versions with the same name.

Runtimes

A Runtime is a Docker image with the predefined gRPC interface which loads and serves your model.

We have implemented a few runtimes, which you can use in your own projects.

Servable

Servable is a deployed instance of a Model version combined with a Runtime. It exposes a gRPC endpoint that can be used to send requests.

Users should not use Servables as-is, since they are designed to be building blocks, rather than inference endpoints. Hydrosphere provides a better alternative to deploy a Model version — Application.

Applications

An Application is a pipeline of one or more stages, each consisting of one or multiple Model Versions. Data sent to an application stage is shadowed to all of its model versions. The output of a stage is picked randomly with respect to weights.

When a user creates an Application, the Manager service automatically deploys appropriate Servables. The Application handles monitoring of your models and can perform A/B traffic splits.

Each Application has publicly available HTTP and gRPC endpoints that you can send requests to.

Deployment Configurations

A Deployment Configuration is a collection of Kubernetes settings that you can set for your Servables and Model Versions used inside of Application stages.

Deployment Configuration covers:

• Horizontal Pod Autoscaler specs

• Container Specs

  • Resource requirements: limits and requests

• Pod Specs

  • Node Selectors

  • Affinity

  • Tolerations

• Deployment Specs

  • Replicas count

Model's Signature

A Model's Signature is a specification of your model computation which identifies the name of a function with its inputs and outputs, including their names, shapes, and data types.

Example of a signature defined in a YAML file:

contract:
  name: predict
  inputs:
    x:
      shape: [-1, 2]
      type: double
      profile: numerical
  outputs:
    y:
      shape: scalar
      type: int
      profile: categorical

Field

A Field is a basic element of a Model's signature. It has a name, shape, data type, and profile.

Example of a model's signature field defined in a YAML file:

x:
  shape: [-1, 2]
  type: double
  profile: numerical

Field`s Profile

A Profile is a special tag that tells how Hydrosphere should interpret the field's data.

There are multiple available tags: Numerical, Categorical, Image, Text, Audio, Video, etc.

Monitoring

Metrics

Data coming through deployed Model Versions can be monitored with metrics.

Metric is a Model Version that takes a combination of inputs & outputs from another monitored Model Version, receives every request and response from the monitored model, produces a single value, and compares it with a threshold to determine whether this request was healthy or not.

Every request is evaluated against all metrics assigned to the model.

Auto OD Metric is an automatically generated Outlier Detection metric. More details are described here.

Checks

A check is a boolean condition associated with a field of a Model Version signature which shows for every request whether the field value is acceptable or not.

For example, Min/Max checks ensure that a field value is in an acceptable range which is inferred from training data values.

Hydrosphere is composed of several microservices, united to efficiently serve and monitor machine learning models in production. Hydrosphere features are divided between multiple services. You can learn more about each of them in this section.

UI / nginx

Serving

Monitoring

Interpretability

Third-Party Integrations

• AWS Sagemaker

Interpretability provides EDA (Exploratory Data Analysis) and explanations for predictions made by your models to make predictions understandable and actionable. It also produces explanations for monitoring metrics to let you know why a particular request was marked as an outlier. The component consists of 2 services:

• Explanations

• Data Projections

Both services are built with Celery to run asynchronous tasks from apps and consists of a client, a worker, and a broker that mediates in between. A client generates a task and initiates it by adding a message to a queue, а broker delivers it to a worker, then the worker executes the task.

Interpretability services use MongoDB as both a Celery broker and backend storage to save task results. To save and retrieve model training and production data, the Interpretability component uses S3 storage.

When Explanation or Data Projection receives a task they create a new temporary Servable specifically for the model they need to make an explanation for. They use this Servable to run data through it in order to make new predictions and delete it after.

Prediction Explanations

Prediction Explanations generate explanations of model predictions to help you understand them. Depending on the type of data your model uses, it provides an explanation as either a set of logical predicates if your data is in a tabular format or a saliency map if your data is in the image format. Saliency Map is a heat map that highlights parts of a picture that a prediction was based on.

Data Projections

Data Projection visualizes high-dimensional data in a 2D scatter plot with an automatically trained UMAP transformer to let you evaluate data structure and spot clusters, outliers, novel data, or any other patterns. It is especially helpful if your model works with high-dimensional data, such as images or text embeddings.

What is Hydrosphere?

Hydrosphere is an open-source MLOps platform for deploying, managing, and monitoring ML models in production with Kubernetes.

Hydrosphere supports all major machine learning frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn, fastai, etc. The platform is designed to effectively measure performance and health metrics of your production models, making it possible to spot early signs of performance drops and data drifts, get insights into why they happen.

Hydrosphere offers immediate value to ML-based products:

• Сovers all aspects of the production ML lifecycle - model versioning & deployment, traffic & contract management, data monitoring, gaining insights.

• Easy & fast management of production models that brings models to production in minutes by reducing time to upload, update, and roll your models into production.

• Allows to create reproducible, observable, and explainable machine learning pipelines.

• Provides understanding and control of models’ performance in production via data and target metrics analysis.

• Adds in-depth observability for your production models and data flowing through them.

• Improves business metrics of ML-based products as a result of a reduction in MTTR and MTTD incidents related to ML models due to early alerts once data drifts happen.

Why Hydrosphere?

Production ML is a dangerous place where numerous things can and usually do go wrong, making issues harder to discover and fix. Hydrosphere automates MLOps in the production part of the ML lifecycle combining best practices of CI/CD and DevOps and putting special emphasis on monitoring performance of ML models after their deployment.

MLOps problems Hydrosphere addresses:

• Non-interpretable, biased models

• Integration between the tools of each step of production ML lifecycle

• Long time to find & debug issues with production ML Models

• Monitoring for Model Degradation and Performance Loss

• Understanding the reasons behind wrong predictions