Skip to main content

Custom Web Server Endpoints

· 3 min read

trainML Endpoints are now even more flexibile with the ability to bring-your-own-web-server.

How It Works​

Endpoints make it easy to expose your model code as a REST API without having to worry about web development or server configuration. However, some applications need the additional flexibility or advanced features that aren't currently available in the built-in API. Now, you can run any web server backend you choose while still utilizing the automated instance management, networking configuration, and certificate management of trainML Endpoints. All you need is a server that provides an HTTP API to listen on port 80, and trainML handles the rest. You can use popular model serving frameworks like Tensorflow Serving or TorchServe, or powerful web servers like NGINX without having to manage infrastructure or networking.

Using the Web Platform​

Navigate to the Endpoint Dashboard and click the Create button. Set the job specifications as you normally would to create the endpoint.

In the Endpoint section, instead of adding Routes, click the Manually Specify Server Command box. Enter the command needed to start the web server in the Start Command field. For example, to run Tensorflow Serving, enter a command similar to the following, replacing the model setting with the location of your models with the job environment.

tensorflow_model_server --rest_api_port=80 \
--model_name="My Model" --model_base_path=${TRAINML_MODEL_PATH}/model
caution

Alternative web servers like Tensorflow Serving or NGINX are not provided in the trainML built-in environments. Either ensure these dependencies will be installed with the dependency installer or use your own custom environment (recommended).

Click Next and Create to start the endpoint. Once the endpoint is running, click the Connect button to view the URL it is accessible on.

tip

If you endpoint stops shortly after starting, check the logs for any execution errors. If none are found, ensure that the start command you use starts the web server in the foreground, not as a background/daemon process. For example, if you are using NGINX, ensure it is configured with daemon off setting.

Using the SDK​

You can also specify the endpoint start command the trainML SDK. Instead of specifying the routes property, in the endpoint dictionary, specify the start_command property. An example specification is the following:

job = await trainml.jobs.create(
    name="Endpoint",
    type="endpoint",
    ...
    endpoint=dict(
        start_command="nginx -g "daemon off;""
    ),
)