Streaming Deployments

Streaming deployments enable you to connect Kafka streams with your models for real-time inference. Leveraging this architecture simplifies the process of processing large amounts of distributed data by eliminating the need for complex triggering and scheduling as fresh data arrives.

Streaming Use Cases

Consider using streaming deployments in the following scenarios:

Deploying Streaming Models

To use streaming inference on JFrog ML, use the following examples:

Option 1: Deploying via UI

Option 2: Deploying via CLI

You can also deploy your streaming model using the command line interface (CLI). Here’s a general command structure:

frogml models deploy stream \
    --model-id <model-id> \
    --build-id <build-id> \
    --instance <instance-type> \
    --replicas <number-of-replicas> \
    --consumer-bootstrap-server <bootstrap-server> \
    --consumer-topic <consumer-topic-name> \
    --producer-bootstrap-server <bootstrap-server> \
    --producer-topic <producer-topic-name>

Example Command 1 (using the above structure): Deploying a FLAN-T5 streaming model with 2 replicas on 2 A10.XL on-demand GPU instances

# Deploy a FLAN-T5 streaming model on 2 A10.XL on-demand instances
frogml models deploy stream \
    --model-id "flan-t5" \
    --build-id "0511ffa9-986e-4fd9-ae29-515269af5bac" \
    --instance "gpu.a10.xl" \
    --replicas 2 \
    --purchase-option "on-demand" \ # Use only on demand instances for executors
    --consumer-bootstrap-server <bootstrap-server> \
    --consumer-topic <consumer-topic-name> \
    --producer-bootstrap-server <bootstrap-server> \
    --producer-topic <producer-topic-name>

Example Command 2 (using the above structure): Deploying a churn streaming model on 2 small instances

# Deploy a churn streaming model on 2 small instances
frogml models deploy stream \
    --model-id "stream-churn-model" \
    --build-id "d4d56c1a-f326-40a0-88f3-f8e59c691a3e" \
    --instance "small" \
    --replicas 2 \
    --consumer-bootstrap-server "10.0.0.8" \
    --consumer-topic "model-input-topic" \
    --producer-bootstrap-server "10.0.0.9" \
    --producer-topic "model-output-topic"

Option 3: Configuring Streaming Models via YAML

You can also configure your streaming model with a YAML file.

Example

build_id: "d4d56c1a-f326-40a0-88f3-f8e59c691a3e"
model_id: "stream-churn-model"
resources:
  instance_size: "small"
  replicas: 2
stream:
  consumer_bootstrap_server: "10.0.0.8"
  consumer_topic: "model-input-topic"
  producer_bootstrap_server: "10.0.0.9"
  producer_topic: "model-output-topic"

Then run the deployment command:

frogml models deploy stream --from-file stream_deploy_config.yaml

Mixing Command Line and Configuration File

Additionally, you can mix and match between command line arguments and configuration file. In this case, command line arguments will take precedence and override the values specified in the YAML file.

Example Command

frogml models deploy stream --replicas 5 --from-file stream_deploy_config.yaml

Monitoring Your Streaming Deployment

JFrog ML models run on Kubernetes, where we automatically install advanced production-grade monitoring tools. This setup provides a robust platform for deploying and managing your models efficiently.

Under the Model Overview tab, you can view the following metrics for your deployed build:

Configuring Streaming Models Overview

JFrog ML streaming deployments enable granular control through multiple consumer and producer specifications. Below is an overview of the key parameters and configurations you can set to tailor your streaming deployment.

Consumer Configuration

Offset Behavior in Redeployments

When you redeploy a streaming model using the same consumer group, the new deployment resumes consumption from the last committed offset for that group.

It does not start consuming messages based on the deployment timestamp. This ensures message continuity—no data is skipped or reprocessed unnecessarily.

If the consumer group does not have a previously committed offset (for example, on the first deployment), the consumer starts from the offset defined by auto.offset.reset:

* Required

Producer Configuration

* Required

Custom Producer and Consumer Configurations

In a streaming inference deployment, you can customize Kafka consumer and producer configurations via environment variables. Use the following prefixes:

For example, to modify the bootstrap.servers configuration for a producer, use the environment variable:

kafka.producer.bootstrap.servers

Example Command for SASL Authentication

If your Kafka topics require SASL authentication, you can provide the necessary username and password as environment variables when deploying the streaming model. The following example shows how to configure both the consumer and producer connections with SASL:

frogml models deploy stream --env-vars \
  kafka.consumer.sasl.mechanism=PLAIN \
  kafka.consumer.security_protocol=SASL_SSL \
  kafka.consumer.sasl_plain_username=<username> \
  kafka.consumer.sasl_plain_password=<password>
  kafka.producer.sasl.mechanism=PLAIN \
  kafka.producer.security_protocol=SASL_SSL \
  kafka.producer.sasl_plain_username=<username> \
  kafka.producer.sasl_plain_password=<password>

Summary

Configuring your streaming deployments with an understanding of replicas, partitions, and their parameters will help you achieve efficient and scalable real-time inference for your applications. Tailor these settings to meet your specific needs for optimal performance and reliability.