Inferencing applications can be achieved using any of the methods described below.
To send a sample request using Hydrosphere UI, open the desired application, and press the Test button at the upper right corner. We will generate dummy inputs based on your model's contract and send an HTTP request to the model's endpoint.
POST /gateway/application/<application_name>
To send an HTTP request, you should send a POST request to the /gateway/application/<applicationName> endpoint with the JSON body containing your request data, composed with respect to the model's contract.
| Name | Type | Description |
|---|---|---|
| Name | Type | Description |
|---|---|---|
To send a gRPC request you need to create a specific client.
You can learn more about our Python SDK here.
application_name
string
Name of the application
object
Request data, composed with respect to the model's contract.
Resource definitions describe Hydrosphere entities.
An entity could be your model, application, or deployment configuration. Each definition is represented by a .yaml file.
Every definition must include the following fields:
kind: defines the type of a resource
name: defines the name of a resource
The only valid options for kind are:
Model
Application
DeploymentConfiguration
A model definition must contain the following fields:
runtime: a string defining the runtime Docker image that will be used to run a model. You can learn more about runtimes here.
contract: an object defining the inputs and outputs of a model.
A model definition can contain the following fields:
payload: a list of files that should be added to the container.
install-command: a string defining a command that should be executed during the container build.
training-data: a string defining a path to the file that will be uploaded to Hydrosphere and used as a training data reference. It can be either a local file or a URI to an S3 object. At the moment we only support .csv files.
metadata: an object defining additional user metadata that will be displayed on the Hydrosphere UI.
The example below shows how a model can be defined on the top level.
contract object must contain the following fields:
inputs: an object, defining all inputs of a model
outputs: an object, defining all outputs of a model
contract object can contain the following fields:
name: a string defining the signature of the model that should be used to process requests
field object must contain the following fields:
shape: either "scalar" or a list of integers, defining the shape of your data. If a shape is defined as a list of integers, it can have -1 value at the very beginning of the list, indicating that this field has an arbitrary number of "entities". -1 cannot be put anywhere aside from the beginning of the list.
type: a string defining the type of data.
field object can contain the following fields:
profile: a string, defining the profile type of your data.
The only valid options for type are:
bool — Boolean
string — String in bytes
half — 16-bit half-precision floating-point
float16 — 16-bit half-precision floating-point
float32 — 32-bit single-precision floating-point
double — 64-bit double-precision floating-point
float64 — 64-bit double-precision floating-point
uint8 — 8-bit unsigned integer
uint16 — 16-bit unsigned integer
uint32 — 32-bit unsigned integer
uint64 — 64-bit unsigned integer
int8 — 8-bit signed integer
int16 — 16-bit signed integer
int32 — 32-bit signed integer
int64 — 64-bit signed integer
qint8 — Quantized 8-bit signed integer
quint8 — Quantized 8-bit unsigned integer
qint16 — Quantized 16-bit signed integer
quint16 — Quantized 16-bit unsigned integer
complex64 — 64-bit single-precision complex
complex128 — 128-bit double-precision complex
The only valid options for profile are:
text — monitoring such fields will be done with text-oriented algorithms.
image — monitoring such fields will be done with image-oriented algorithms.
numerical — monitoring such fields will be done with numerical-oriented algorithms.
categorical — monitoring such fields will be done with categorical-oriented algorithms.
The example below shows how a contract can be defined on the top level.
metadata object can represent any arbitrary information specified by the user. The structure of the object is not strictly defined. The only constraint is that the object must have a key-value structure, where a value can only be of a simple data type (string, number, boolean).
The example below shows, how metadata can be defined.
The example below shows a complete definition of a sample model.
The application definition must contain one of the following fields:
singular: An object, defining a single-model application;
pipeline: A list of objects, defining an application as a pipeline of models.
singular object represents an application consisting only of one model. The object must contain the following fields:
model: A string, defining a model version. It is expected to be in the form model-name:model-version.
The example below shows how a singular application can be defined.
pipeline represents a list of stages, representing models.
stage object must contain the following fields:
model: A string defining a model version. It is expected to be in the form model-name:model-version.
stage object can contain the following fields:
weight: A number defining the weight of the model. All models' weights in a stage must add up to 100.
The example below shows how a pipeline application can be defined.
In this application, 100% of the traffic will be forwarded to the claims-preprocessing:1 model version and the output will be fed into claims-model. 80% of the traffic will go to the claims-model:1 model version, 20% of the traffic will go to the claims-model:2 model version.
The DeploymentConfiguration resource definition can contain the following fields:
hpa: An object defining HorizontalPodAutoscalerSpec
container: An object defining settings applied on a container level
deployment: An object defining settings applied on a deployment level
pod: An object defining settings applied on a pod level
The hpa object closely resembles the Kubernetes HorizontalPodAutoscalerSpec object
The hpa object must contain:
minReplicas : minReplicas is the lower limit for the number of replicas to which the autoscaler can scale down.
maxReplicas : integer, upper limit for the number of pods that can be set by the autoscaler; cannot be smaller than minReplicas.
cpuUtilization : integer from 1 to 100, target average CPU utilization (represented as a percentage of requested CPU) over all the pods; if not specified the default autoscaling policy will be used.
The container object can contain:
resources : object with limits and requests fields. Closely resembles the k8s ResourceRequirements object
env : object with string keys and string values which is used to set environment variables.
The hpa object is similar to the Kubernetes PodSpec object.
The pod object can contain
affinity : pod's scheduling constraints. Represented by an Affinity object.
tolerations : array of Tolerations.
The deployment object must contain:
replicaCount : integer, number of desired pods. This is a pointer to distinguish between explicit zero and not specified. Defaults to 1.
The example below shows how a deployment configuration can be defined.
To use private pip repository you must add customized pip.conf file pointing to your custom PyPI repository.
For example, your custom pip.conf file can look like this:
If you need to specify the certificate to use during pip install you want to specify the path to it in a pip.conf file e.g.
You can tell pip to use this pip.conffile in the install-command field inside serving.yaml:
Sometimes our runtime images are not flexible enough. In that case, you might want to implement one yourself.
The key things you need to know to write your own runtime are:
How to implement a predefined gRPC service for a dedicated language
How to our contracts' protobufs work to describe entry points, such as inputs and outputs
How to create your own Docker image and publish it to an open registry
There are different approaches to generating client and server gRPC code in different languages. Let's have a look at how to do that in Python.
First, let's clone our protos repository and prepare a folder for the generated code:
To generate the gRPC code we need to install additional packages:
Our custom runtime will require contracts and tf protobuf messages. Let's generate them:
The structure of the runtime should now be as follows:
Now that we have everything set up, let's implement a runtime. Create a runtime.py file and put in the following code:
Let's quickly review what we have here. RuntimeManager simply manages our service, i.e. starts it, stops it, and holds all necessary data. RuntimeService is a service that actually implements thePredict(PredictRequest) RPC function.
The model will be stored inside the /model directory in the Docker container. The structure of /model is a follows:
Thecontract.protobin file will be created by the Manager service. It contains a binary representation of the ModelContract message.
files directory contains all files of your model.
To run this service let's create an another file main.py.
Before we can use the runtime, we have to package it into a container.
To add requirements for installing dependencies, create a requirements.txt file and put inside:
Create a Dockerfile to build our image:
APP_PORT is an environment variable used by Hydrosphere. When Hydrosphere invokes Predict method, it does so via the defined port.
The structure of the runtime folder should now look like this:
Build and push the Docker image:
Remember that the registry has to be accessible to the Hydrosphere platform so it can pull the runtime whenever it has to run a model with this runtime.
That's it. You have just created a simple runtime that you can use in your own projects. It is an almost identical version of our python runtime implementation. You can always look up details there.
This section offers guides that address technical aspects of working with the Hydrosphere platform.