Real-Time Deployments

JFrog ML real time models deploy your ML models with a lightweight, simple and scalable REST API wrapper.

We set up the network requirements and deploy your model on a managed Kubernetes cluster, enabling you to leverage auto-scaling and security. JFrog ML also adds a suite of monitoring tools, simplifying the process of managing your model performance.

Deploying Real-time Models from the UI

To deploy a real-time model from the UI:

Deploying Real-time Models from the CLI

To deploy a model in real-time mode from the CLI, populate the following command template:

frogml models deploy realtime \
    --model-id <model-id> \
    --build-id <build-id> \
    --pods <pods-count> \
    --instance <instance-type> \
    --timeout <timeout-ms> \
    --server-workers <workers> \
    --variation-name <variation-name> \
    --daemon-mode <bool>

For example, for the model built in the Get Started with JFrog ML section, the deployment command is:

frogml models deploy realtime \
    --model-id churn_model \
    --build-id 7121b796-5027-11ec-b97c-367dda8b746f \
    --pods 2 \
    --instance small \
    --timeout 3000 \
    --server-workers 4 \
    --variation-name default \
    --daemon-mode false

Deploying a Real-time Model Using GPUs

Realtime models can be deployed on GPU instances, simply by selecting a GPU Instance from the available options.

frogl models deploy realtime \
    --model-id churn_model \
    --build-id 7121b796-5027-11ec-b97c-367dda8b746f \
    --pods 4 \
    --instance gpu.a10.xl \
    --timeout 3000 \
    --server-workers 4 \
    --variation-name default \
    --daemon-mode false

Configuring Real-time Models

The following table contains the possible parameters and variables for deploying real-time models.

Using Custom AWS IAM Role

In some cases, a model needs to access external services during the runtime. If your model requires access to AWS resources, a custom AWS IAM role can be passed during the deployment process.

The IAM role should be created with the following trust policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::<account-id>:root"
      },
      "Action": "sts:AssumeRole",
      "Condition": {
        "ArnLike": {
          "aws:PrincipalArn": "arn:aws:iam::<account-id>:role/qwak-eks-base*"
        }
      }
    }
  ]
}

The IAM role ARN can be passed directly to a deployment using the --iam-role-arn flag. For example:

frogml models deploy realtime \
    --model-id churn_model \
    --build-id 7121b796-5027-11ec-b97c-367dda8b746f \
    --pods 4 \
    --instance small \
    --timeout 3000 \
    --server-workers 4 \
    --variation-name default \
    --daemon-mode false \
    --iam-role-arn arn:aws:iam::<account-id>:role/<role-name>

Deploying Real-time Models Locally

To run the deployment locally using a local Docker engine, use the --local flag. For example:

frogml models deploy realtime \
    --model-id churn_model \
    --build-id 7121b796-5027-11ec-b97c-367dda8b746f \
    --local

Real-time Model Inference

Once you have successfully deployed a real-time model, you can use the JFrog ML Inference SDK to perform invocations.

In this example, we'll invoke a model via the Python SDK, which can be easily installed using:

pip install frogml-inference

Model inference parameters are model specific.

For the below model, assuming it was built and deployed successfully as a real-time endpoint, with the model ID iris_classifier:

from frogml import api, FrogMlModel
from sklearn import svm, datasets
import pandas as pd

class IrisClassifier(FrogMlModel):

    def __init__(self):
        self._gamma = 'scale'
        self._model = None

    def build(self):
        # load training data
        iris = datasets.load_iris()
        X, y = iris.data, iris.target

        # Model Training
        clf = svm.SVC(gamma=self._gamma)
        self._model = clf.fit(X, y)

    @api()
    def predict(self, df: pd.DataFrame) -> pd.DataFrame:
        return pd.DataFrame(data=self._model.predict(df), columns=['species'])

A prediction call from the JFrog ML Python SDK is:

from frogml_inference import RealTimeClient

model_id = "iris_classifier"
feature_vector = [
   {
      "sepal_width": 3,
      "sepal_length": 3.5,
      "petal_width": 4,
      "petal_length": 5
   }]

client = RealTimeClient(model_id=model_id)
response = client.predict(feature_vector)

Monitoring Real-time Endpoints

JFrog ML endpoints are deployed on Kubernetes, coupled with advanced monitoring tools for production-grade readiness.

JFrog ML comes bundled with Grafana and Prometheus to provide monitoring dashboards, and ElasticSearch for log collection, amongst other tools.

The following health metrics appear in the model Overview tab:

In addition, you can follow and search the applicable logs produced by your model in the Logs tab:

Auto-scaling Real-time Models

To attach a new auto scaling to a running model:

1. Create a config file:

api_version: v1
spec:
  model_id: <model-id>
  variation_name: <variation-name>
  auto_scaling:
    min_replica_count: 1
    max_replica_count: 10
    polling_interval: 30
    cool_down_period: 300
    triggers:
      prometheus_trigger:
        - query_spec:
            metric_type: <cpu/gpu/memory/latency/error_rate/throughput>
            aggregation_type: <min/max/avg/sum>
            time_period: 30
          threshold: 60

2. Run the following command:

frogml models autoscaling attach -f config.yaml

Configuration