Use Models in Your Code

This procedure explains how to utilize the allowed models within your organization's code.

Note

Note that the integration process varies depending on the model type. Follow the procedure according to the model that you are invoking.

External API Models

To use an allowed external API model:

When a model is allowed, the Use Model button appears in the Model information window with the relevant code snippets.

In the Registry page, select the project on the left.
Select the allowed model under the selected project.
The model dashboard is shown.
For details on how to read the dashboard to monitor and interpret the model's performance, refer to Use the Model Dashboards
Note that the dashboard only shows data when there is traffic.
Click Use Model. The Use Model pane is displayed. Here you are able to generate a token and insert it into the code snippet for the framework of your choice.

Browse through the example code snippets for different libraries and frameworks (Python, Javascript, cURL).

Note that this code snippet shown includes a placeholder for the api_key for the token you are about to generate, and the model name, which includes the name of the connection.

from openai import OpenAI

client = OpenAI(
  api_key="your_jfrog_api_key",
  base_url="https://models.a0smkltuxjyst.qwak.ai/v1"
)

response = client.chat.completions.create(
  model="OpenAI/gpt-3.5-turbo-1106",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {
      "role": "user",
      "content": "Explain to me how AI works in one sentence"
    }
  ]
)

print(response.choices[0].message)

Click Generate a token. The Set Up A Client pane is displayed.
Note
Keep the default repository.
In the Configure tab, enter your JFrog account password and click Generate Token & Create Instructions. The token is displayed.
Click Copy.
Click Done.

Select the correct framework (Python/Javascript/cURL), and note that the token has been inserted into the api_key.

Example Python

from openai import OpenAI

client = OpenAI(
  api_key="abcdefghijklmnopqrstuvwxyz1234567890_tokenexample",
  base_url="https://models.a0smkltuxjyst.qwak.ai/v1"
)

response = client.chat.completions.create(
  model="Gemini/gemini-1.5-flash",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {
      "role": "user",
      "content": "Explain to me how AI works in one sentence"
    }
  ]
)

print(response.choices[0].message)

Copy the code snippet into your code editor. You can now use this model in your applications.

Model Packages

After a model package has been allowed, the Use Model and Deploy buttons appear in the Model information window, enabling you to either use the model with the transformers library or to use it in all your applications.

Using the Model with Your Transformers Library vs. Deploying:

Use Model: Enables you to integrate the model with the transformers library, you use the model locally within your development environment. This approach is ideal for experimenting, testing, and small-scale applications. For instructions, see below.
Deploy: Deploying the model through this platform involves configuring it to run within a managed infrastructure. Once deployed, the model can be accessed and used across all your applications, facilitating consistent performance and scalability. For instructions, see Model Packages.

To use the allowed model only with the transformers library:

Click the allowed model.
Click the project name in the Allowed in projects list.
Click Use Model. The Use model pane is displayed. Here you define how to use this model securely with your framework of choice.
Note
Keep the default repository.
Click Configure to configure your SDK to pull this model. The Set Up A HuggingFaceML Client pane is displayed. These steps tell the system where to take the model from.
Enter your JFrog account password and click Generate Token & Create Instructions.
Follow the instructions on the Configure tab:
1. Copy the code snippet with the export commands into your code interface. This configures the Hugging Face client to work with Artifactory, and adds your repository.
```
export HF_HUB_ETAG_TIMEOUT=86400
export HF_HUB_DOWNLOAD_TIMEOUT=86400
export HF_ENDPOINT=https://jmlsoleng.jfrog.io/artifactory/api/huggingfaceml/research-research-huggingface-remote
```
2. Copy and run the code snippet with the export token (to authenticate the Hugging Face client with Artifactory).
```
export HF_TOKEN=tokenexample
```
3. Click Done.

To use the allowed model, please copy the Transformers code snippet into your Python environment:

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained("model_name")
model = AutoModelForCausalLM.from_pretrained("model_name")

Now you are ready to use the model directly within your development environment using the Transformers library.

To use the model with all your applications (Deploy):

See Deploy Model Packages for instructions. These instructions include how to deploy the model and also how to configure the model for use in your developments. After you have deployed and configured the model, you can see the model's usage metrics in the model Overview page.