Develop runtimes
Sometimes our runtime images are not flexible enough. In that case, you might want to implement one yourself.
The key things you need to know to write your own runtime are:
- How to implement a predefined gRPC service for a dedicated language
- How to our contracts' protobufs work to describe entry points, such as inputs and outputs
- How to create your own Docker image and publish it to an open registry
Generate GRPC code
There are different approaches to generating client and server gRPC code in different languages. Let's have a look at how to do that in Python.
1
$ git clone https://github.com/Hydrospheredata/hydro-serving-protos
2
$ mkdir runtime
Copied!
To generate the gRPC code we need to install additional packages:
1
$ pip install grpcio-tools googleapis-common-protos
Copied!
Our custom runtime will require
contracts and tf protobuf messages. Let's generate them:1
$ python -m grpc_tools.protoc --proto_path=./hydro-serving-protos/src/ --python_out=./runtime/ --grpc_python_out=./runtime/ $(find ./hydro-serving-protos/src/hydro_serving_grpc/contract/ -type f -name '*.proto')
2
$ python -m grpc_tools.protoc --proto_path=./hydro-serving-protos/src/ --python_out=./runtime/ --grpc_python_out=./runtime/ $(find ./hydro-serving-protos/src/hydro_serving_grpc/tf/ -type f -name '*.proto')
3
$ cd runtime
4
$ find ./hydro_serving_grpc -type d -exec touch {}/__init__.py \;
Copied!
The structure of the
runtime should now be as follows:1
runtime
2
└── hydro_serving_grpc
3
├── __init__.py
4
├── contract
5
│ ├── __init__.py
6
│ ├── model_contract_pb2.py
7
│ ├── model_contract_pb2_grpc.py
8
│ ├── model_field_pb2.py
9
│ ├── model_field_pb2_grpc.py
10
│ ├── model_signature_pb2.py
11
│ └── model_signature_pb2_grpc.py
12
└── tf
13
├── __init__.py
14
├── api
15
│ ├── __init__.py
16
│ ├── model_pb2.py
17
│ ├── model_pb2_grpc.py
18
│ ├── predict_pb2.py
19
│ ├── predict_pb2_grpc.py
20
│ ├── prediction_service_pb2.py
21
│ └── prediction_service_pb2_grpc.py
22
├── tensor_pb2.py
23
├── tensor_pb2_grpc.py
24
├── tensor_shape_pb2.py
25
├── tensor_shape_pb2_grpc.py
26
├── types_pb2.py
27
└── types_pb2_grpc.py
Copied!
Implement Service
Now that we have everything set up, let's implement a runtime. Create a
runtime.py file and put in the following code:1
from hydro_serving_grpc.tf.api.predict_pb2 import PredictRequest, PredictResponse
2
from hydro_serving_grpc.tf.api.prediction_service_pb2_grpc import PredictionServiceServicer, add_PredictionServiceServicer_to_server
3
from hydro_serving_grpc.tf.types_pb2 import *
4
from hydro_serving_grpc.tf.tensor_pb2 import TensorProto
5
from hydro_serving_grpc.contract.model_contract_pb2 import ModelContract
6
from concurrent import futures
7
8
import os
9
import time
10
import grpc
11
import logging
12
import importlib
13
14
15
class RuntimeService(PredictionServiceServicer):
16
def __init__(self, model_path, contract):
17
self.contract = contract
18
self.model_path = model_path
19
self.logger = logging.getLogger(self.__class__.__name__)
20
21
def Predict(self, request, context):
22
self.logger.info(f"Received inference request: {request}")
23
24
module = importlib.import_module("func_main")
25
executable = getattr(module, self.contract.predict.signature_name)
26
result = executable(**request.inputs)
27
28
if not isinstance(result, hs.PredictResponse):
29
self.logger.warning(f"Type of a result ({result}) is not `PredictResponse`")
30
context.set_code(grpc.StatusCode.OUT_OF_RANGE)
31
context.set_details(f"Type of a result ({result}) is not `PredictResponse`")
32
return PredictResponse()
33
return result
34
35
36
class RuntimeManager:
37
def __init__(self, model_path, port):
38
self.logger = logging.getLogger(self.__class__.__name__)
39
self.port = port
40
self.model_path = model_path
41
self.server = None
42
43
with open(os.path.join(model_path, 'contract.protobin')) as file:
44
contract = ModelContract.ParseFromString(file.read())
45
self.servicer = RuntimeService(os.path.join(self.model_path, 'files'), contract)
46
47
def start(self):
48
self.logger.info(f"Starting PythonRuntime at {self.port}")
49
self.server = grpc.server(futures.ThreadPoolExecutor(max_workers=10))
50
add_PredictionServiceServicer_to_server(self.servicer, self.server)
51
self.server.add_insecure_port(f'[::]:{self.port}')
52
self.server.start()
53
54
def stop(self, code=0):
55
self.logger.info(f"Stopping PythonRuntime at {self.port}")
56
self.server.stop(code)
57
© 2020 GitHub, Inc.
Copied!
Let's quickly review what we have here.
RuntimeManager simply manages our service, i.e. starts it, stops it, and holds all necessary data. RuntimeService is a service that actually implements thePredict(PredictRequest) RPC function.The model will be stored inside the
/model directory in the Docker container. The structure of /model is a follows:1
model
2
├── contract.protobin
3
└── files
4
├── ...
5
└── ...
Copied!
The
contract.protobin file will be created by the Manager service. It contains a binary representation of the ModelContract message.files directory contains all files of your model.To run this service let's create an another file
main.py.1
from runtime import RuntimeManager
2
3
import os
4
import time
5
import logging
6
7
logging.basicConfig(level=logging.INFO)
8
9
if __name__ == '__main__':
10
runtime = RuntimeManager('/model', port=int(os.getenv('APP_PORT', "9090")))
11
runtime.start()
12
13
try:
14
while True:
15
time.sleep(60 * 60 * 24)
16
except KeyboardInterrupt:
17
runtime.stop()
Copied!
Publish Runtime
Before we can use the runtime, we have to package it into a container.
To add requirements for installing dependencies, create a
requirements.txt file and put inside:1
grpcio==1.12.1
2
googleapis-common-protos==1.5.3
Copied!
Create a Dockerfile to build our image:
1
FROM python:3.6.5
2
3
ADD . /app
4
RUN pip install -r /app/requirements.txt
5
6
ENV APP_PORT=9090
7
8
VOLUME /model
9
WORKDIR /app
10
11
CMD ["python", "main.py"]
Copied!
APP_PORT is an environment variable used by Hydrosphere. When Hydrosphere invokes Predict method, it does so via the defined port.The structure of the
runtime folder should now look like this:1
runtime
2
├── Dockerfile
3
├── hydro_serving_grpc
4
│ ├── __init__.py
5
│ ├── contract
6
│ │ ├── __init__.py
7
│ │ ├── model_contract_pb2.py
8
│ │ ├── model_contract_pb2_grpc.py
9
│ │ ├── model_field_pb2.py
10
│ │ ├── model_field_pb2_grpc.py
11
│ │ ├── model_signature_pb2.py
12
│ │ └── model_signature_pb2_grpc.py
13
│ └── tf
14
│ ├── __init__.py
15
│ ├── api
16
│ │ ├── __init__.py
17
│ │ ├── model_pb2.py
18
│ │ ├── model_pb2_grpc.py
19
│ │ ├── predict_pb2.py
20
│ │ ├── predict_pb2_grpc.py
21
│ │ ├── prediction_service_pb2.py
22
│ │ └── prediction_service_pb2_grpc.py
23
│ ├── tensor_pb2.py
24
│ ├── tensor_pb2_grpc.py
25
│ ├── tensor_shape_pb2.py
26
│ ├── tensor_shape_pb2_grpc.py
27
│ ├── types_pb2.py
28
│ └── types_pb2_grpc.py
29
├── main.py
30
├── requirements.txt
31
└── runtime.py
Copied!
Build and push the Docker image:
1
$ docker build -t {username}/python-runtime-example
2
$ docker push {username}/python-runtime-example
Copied!
Remember that the registry has to be accessible to the Hydrosphere platform so it can pull the runtime whenever it has to run a model with this runtime.
That's it. You have just created a simple runtime that you can use in your own projects. It is an almost identical version of our python runtime implementation. You can always look up details there.
Last modified 9mo ago
Export as PDF
Copy link