Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Loading...
Hydrosphere has an internal Model Registry as centralized storage for Model Versions. When you build a Dockerized model and upload it to Hydrosphere or create new model versions, they get uploaded/stored to the configured model registry in the form of images. This organizes and simplifies model management across the platform and production lifecycle.
Gateway is a service responsible for routing requests to/from or between Servables and Applications and validating these requests for matching a Model's/Application signature.
The Gateway maps a model’s name to a corresponding container. Whenever it receives a request via HTTP API, GRPC, or Kafka Streams, it communicates with that container via the gRPC protocol.
Manager is responsible for:
Building a Docker Image from your ML model for future deployment
Storing these images inside a Docker Registry deployed alongside with
manager service
Versioning these images as Model Versions
Creating running instances of these Model Versions called Servables
inside Kubernetes cluster
Combining multiple Model Versions into a linear graph with a single
endpoint called Application
Platform for deploying your Machine Learning to production
Hydrosphere is a platform for deploying, versioning, and monitoring your machine learning models in production. It is language-agnostic and framework-agnostic, with support for all major programming languages and frameworks - Python, Java, Tensorflow, Pytorch, etc.
⭐️ Star Hydrosphere repo on Github
💦 Explore our Getting Started tutorial
🥳 Join Hydrosphere Slack Community
Sonar service is responsible for managing metrics, training and production data storage, calculating profiles, and shadowing data to the Model Versions which are used as an outlier detection metrics.
There are a few concepts that you should be familiar with before starting to work with the Hydrosphere platform.
Resource definitions describe Models, Applications, and Deployment Configurations in the YAML format. You can learn more about them in the How to write resource definitions section.
A Model is a machine learning model or a processing function that consumes provided inputs and produces predictions or transformations.
Within the Hydrosphere platform, we break down a model into its versions. Each Model version represents a single Docker image containing all the artifacts that you have uploaded to the platform. Consequently, Model is a group of Model versions with the same name.
A Runtime is a Docker image with the predefined gRPC interface which loads and serves your model.
We have implemented a few runtimes, which you can use in your own projects.
Servable is a deployed instance of a Model version combined with a Runtime. It exposes a gRPC endpoint that can be used to send requests.
Users should not use Servables as-is, since they are designed to be building blocks, rather than inference endpoints. Hydrosphere provides a better alternative to deploy a Model version — Application.
An Application is a pipeline of one or more stages, each consisting of one or multiple Model Versions. Data sent to an application stage is shadowed to all of its model versions. The output of a stage is picked randomly with respect to weights.
When a user creates an Application, the Manager service automatically deploys appropriate Servables. The Application handles monitoring of your models and can perform A/B traffic splits.
Each Application has publicly available HTTP and gRPC endpoints that you can send requests to.
A Deployment Configuration is a collection of Kubernetes settings that you can set for your Servables and Model Versions used inside of Application stages.
Deployment Configuration covers:
Horizontal Pod Autoscaler specs
Container Specs
Resource requirements: limits and requests
Pod Specs
Node Selectors
Affinity
Tolerations
Deployment Specs
Replicas count
A Model's Signature is a specification of your model computation which identifies the name of a function with its inputs and outputs, including their names, shapes, and data types.
Example of a signature defined in a YAML file:
A Field is a basic element of a Model's signature. It has a name, shape, data type, and profile.
Example of a model's signature field defined in a YAML file:
A Profile is a special tag that tells how Hydrosphere should interpret the field's data.
There are multiple available tags: Numerical, Categorical, Image, Text, Audio, Video, etc.
Data coming through deployed Model Versions can be monitored with metrics.
Metric is a Model Version that takes a combination of inputs & outputs from another monitored Model Version, receives every request and response from the monitored model, produces a single value, and compares it with a threshold to determine whether this request was healthy or not.
Every request is evaluated against all metrics assigned to the model.
Auto OD Metric is an automatically generated Outlier Detection metric. More details are described here.
A check is a boolean condition associated with a field of a Model Version signature which shows for every request whether the field value is acceptable or not.
For example, Min/Max checks ensure that a field value is in an acceptable range which is inferred from training data values.
Interpretability provides EDA (Exploratory Data Analysis) and explanations for predictions made by your models to make predictions understandable and actionable. It also produces explanations for monitoring metrics to let you know why a particular request was marked as an outlier. The component consists of 2 services:
Explanations
Data Projections
Both services are built with Celery to run asynchronous tasks from apps and consists of a client, a worker, and a broker that mediates in between. A client generates a task and initiates it by adding a message to a queue, а broker delivers it to a worker, then the worker executes the task.
Interpretability services use MongoDB as both a Celery broker and backend storage to save task results. To save and retrieve model training and production data, the Interpretability component uses S3 storage.
When Explanation or Data Projection receives a task they create a new temporary Servable specifically for the model they need to make an explanation for. They use this Servable to run data through it in order to make new predictions and delete it after.
Prediction Explanations generate explanations of model predictions to help you understand them. Depending on the type of data your model uses, it provides an explanation as either a set of logical predicates if your data is in a tabular format or a saliency map if your data is in the image format. Saliency Map is a heat map that highlights parts of a picture that a prediction was based on.
Data Projection visualizes high-dimensional data in a 2D scatter plot with an automatically trained UMAP transformer to let you evaluate data structure and spot clusters, outliers, novel data, or any other patterns. It is especially helpful if your model works with high-dimensional data, such as images or text embeddings.
Hydrosphere is an open-source MLOps platform for deploying, managing, and monitoring ML models in production with Kubernetes.
Hydrosphere supports all major machine learning frameworks, including Tensorflow, Keras, PyTorch, XGBoost, scikit-learn, fastai, etc. The platform is designed to effectively measure performance and health metrics of your production models, making it possible to spot early signs of performance drops and data drifts, get insights into why they happen.
Hydrosphere offers immediate value to ML-based products:
Сovers all aspects of the production ML lifecycle - model versioning & deployment, traffic & contract management, data monitoring, gaining insights.
Easy & fast management of production models that brings models to production in minutes by reducing time to upload, update, and roll your models into production.
Allows to create reproducible, observable, and explainable machine learning pipelines.
Provides understanding and control of models’ performance in production via data and target metrics analysis.
Adds in-depth observability for your production models and data flowing through them.
Improves business metrics of ML-based products as a result of a reduction in MTTR and MTTD incidents related to ML models due to early alerts once data drifts happen.
Production ML is a dangerous place where numerous things can and usually do go wrong, making issues harder to discover and fix. Hydrosphere automates MLOps in the production part of the ML lifecycle combining best practices of CI/CD and DevOps and putting special emphasis on monitoring performance of ML models after their deployment.
MLOps problems Hydrosphere addresses:
Non-interpretable, biased models
Integration between the tools of each step of production ML lifecycle
Long time to find & debug issues with production ML Models
Monitoring for Model Degradation and Performance Loss
Understanding the reasons behind wrong predictions