Define Automations

JFrog ML offers built-in automations for various recurring actions in the model lifecycle. These automations are meant to replace external orchestration tools such as Airflow for example.

In the JFrog ML platform, you can easily define automations for model retraining or batch model executions on either time based or trigger based.

Configuring Automations

We now create an instance of the Automation class and configure it:

from frogml.core.automations import Automation, ScheduledTrigger, FrogmlBuildDeploy,\
    BuildSpecifications, BuildMetric, ThresholdDirection, DeploymentSpecifications

test_automation = Automation(
    name="retrain_my_model",
    model_id="my-model-id",
    trigger=ScheduledTrigger(cron="0 0 * * 0"),
    action=FrogmlBuildDeploy(
        build_spec=BuildSpecifications(git_uri="https://github.com/org_id/repository_name.git#dir_1/dir_2",
                                       git_access_token_secret="token_secret_name",
                                       git_branch="main",
                                       main_dir="main",
                                       tags=["prod"],
                                       env_vars=["key1=val1", "key2=val2", "key3=val3"]),
        deployment_condition=BuildMetric(metric_name="f1_score",
                                         direction=ThresholdDirection.ABOVE,
                                         threshold="0.65"),
        deployment_spec=DeploymentSpecifications(number_of_pods=1,
                                                 cpu_fraction=2.0,
                                                 memory="2Gi",
                                                 variation_name="B")
    )
)

Selecting an Instance Type

You can specify the purchase_option parameter of the BuildSpecifications to select between on-demand and spot instances. Available values: spot and ondemand. For example:

... 
build_spec=BuildSpecifications(
  git_uri="https://github.com/org_id/repository_name.git#dir_1/dir_2",
  git_access_token_secret="token_secret_name",
  git_branch="main",
  main_dir="main",
  tags=["prod"],
  env_vars=["key1=val1", "key2=val2", "key3=val3"],
  purchase_option="ondemand"
),
 ...

spot is the default value if you don't specify anything.

Triggering Automations

Automations may be configured to either compare the model's performance with a pre-defined threshold and deploy the model when the evaluation results pass. 

We can trigger the automation in two ways: 

Schedule-based Triggers

Triggering an automation based on an interval name or a cron expression.

In the case of an interval configuration, the trigger configuration would look like this:

from frogml.core.automations import ScheduledTrigger

# Valid values: Daily, Weekly, Hourly
ScheduledTrigger(interval="Daily")

Metric-based Triggers

To retrain the model based on production performance metrics, we can use the MetricBasedTrigger.

In this case, we need to specify a SQL query which should return the metric value from JFrog ML model Analytics:

from frogml.core.automations import MetricBasedTrigger, ThresholdDirection, SqlMetric

MetricBasedTrigger(
    name='metric_name',
    metric=SqlMetric(sql_query='SQL_QUERY'),
    direction=ThresholdDirection.ABOVE,
    threshold="0.7"
)

Notifications

JFrog ML supports configuring notifications in case of an error or success using either Slack Webhooks or a custom webhook configuration.

Slack Webhook

from frogml.core.automations import SlackNotification, ScheduledTrigger,FrogmlBuildDeploy,\
    BuildSpecifications, BuildMetric, DeploymentSpecifications, Automation

test_automation = Automation(
    name="my_automation",
    model_id="my_model",
    trigger=ScheduledTrigger(cron="0 0 * * 0"),
    action=FrogmlBuildDeploy(
        build_spec=BuildSpecifications(...),
        deployment_condition=BuildMetric(...),
        deployment_spec=DeploymentSpecifications(...)
    ),
    on_error=SlackNotification(webhook="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX"),
    on_success=SlackNotification(webhook="https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX")
)

It will send a slack message when the automation is finished - containing the execution time, the model, the automation name, and the final status of the automation.

Custom Webhook

Another option is to be notified to a custom webhook, using the following configuration:

from frogml.core.automations import CustomWebhook

on_success=CustomWebhook(
    url="http://my.api/endpoint/v1",
    http_method="GET", # defaults to "GET"
    data={"a1": "b1", "a2": "b2"},
    headers={"content-type": "application/json"}
)

The definition above will call the API (defined by the url) above with the following parameters:

In addition to the fields explicitly defined in the data field.

If the HTTP method defined is GET - the data field plus the JFrog ML parameters above will be embedded as request parameters. In all other methods - in the body to attached to the request. 

Registering the Automation

Now, we can register the automation using the JFrog ML CLI:

frogml automations register -p .

In the command above, we specified the directory containing the automation definitions (-p). In this case, the current working directory.