Resolve NVIDIA NIM Models

JFrog Artifactory Documentation

Products

JFrog Artifactory

Content Type

User Guide

ft:sourceType

Paligo

This topic describes how to resolve NIM Models using NIM remote repositories in Artifactory. It provides instructions to resolve packages using NIM remote repository that points to https://api.ngc.nvidia.com.

To run the NIM model Docker container, execute the following command in your terminal.

The sample snippet structure is as follows:

Note

Make sure to replace the placeholders in bold with appropriate values.

The following fields require your input:

[docker-repository]: The NIM docker remote repository key. The Docker image for the model will be pulled from the Docker repository.
[image-path]: The path of the NIM image
[tag]: The tag of the NIM image

docker run -it --rm --gpus all \
   -e NGC_API_KEY="$NGC_API_KEY" \
   -e NGC_API_ENDPOINT="$NGC_API_ENDPOINT" \
   -e NGC_API_SCHEME="$NGC_API_SCHEME" \
   -e NGC_AUTH_ENDPOINT="$NGC_AUTH_ENDPOINT" \
   -v "$CACHE_DIR:/opt/nim/.cache" \
   [<docker-repository>]/[<image-path>]:[<tag>]

This docker run command runs a Docker container with various configurations and parameters. Here's a breakdown of each parameter:

docker run: This is the base command to run a Docker container. It starts the container, and you can specify various options to configure the environment and behavior.
-it:
- -i stands for interactive, which keeps the standard input (stdin) open, allowing you to interact with the container if needed.
- -t allocates a pseudo-TTY, which provides terminal features like color and text formatting inside the container.
Together, -it allows for interactive use of the container in the terminal (for example, if you want to run commands inside it).
--rm: This option automatically removes the container once it exits. It is useful for keeping your system clean by ensuring that stopped containers don't remain on your system.
--gpus all: This option allows Docker to utilize all available GPUs on your machine. The container will have access to all the GPUs on the host system (assuming the Docker daemon supports GPU usage, typically with NVIDIA Docker or CUDA support).
-e NGC_API_KEY="$NGC_API_KEY": This sets an environment variable NGC_API_KEY inside the container. The value of NGC_API_KEY is taken from the host system's environment variable $NGC_API_KEY. This is commonly used to provide API keys or secrets.
-e NGC_API_ENDPOINT="$NGC_API_ENDPOINT": This sets an environment variable, NGC_API_ENDPOINT, inside the container. The value is taken from the host system's $NGC_API_ENDPOINT. It might be used to set the endpoint for some API interactions (for example, a cloud service or registry).
-e NGC_API_SCHEME="$NGC_API_SCHEME": This sets an environment variable NGC_API_SCHEME inside the container. The value is taken from the host system's $NGC_API_SCHEME.
-e NGC_AUTH_ENDPOINT="$NGC_AUTH_ENDPOINT": This sets the NGC_AUTH_ENDPOINT environment variable to the value of NGC_AUTH_ENDPOINT.
-v "$CACHE_DIR:/opt/nim/.cache": This binds a volume (shared directory) between the host and the container. The host's directory "$CACHE_DIR" is mapped to /opt/nim/.cache inside the container. This allows data or files to be shared between the host and container, or to persist data outside the container after it exits.
- $CACHE_DIR is an environment variable from the host that represents the path to a cache directory.
[<docker-repository>]/[<image-path>]:[<tag>]: This specifies the Docker image to run. It consists of three parts:
- <docker-repository>: The specified Docker repository from which the model will be pulled.
- <image-path>: The specific path to the image within the repository. For example, nim/meta/llama-3.1-8b-instruct.
- <tag>: The version or tag of the Docker image. This could be a version number, such as the latest, 1.1.0, or a specific commit ID.
To get the container image and tag, follow these steps:
1. From the NGC Catalog, click Container.
2. Search for your desired container, locate it, and then click on the container.
3. Click Get Container and copy the image path (the image path refers to the part of the string that starts after nvcr.io/ and ends just before the :)
  For example, nim/meta/llama-3.1-8b-instruct
4. To view all tags, click View all tags.
For example: If you wanted to run a llama Docker image with GPU support, it could look like
```
docker run -it --rm --gpus all \
   -e NGC_API_KEY="$NGC_API_KEY" \
   -e NGC_API_ENDPOINT="$NGC_API_ENDPOINT" \
   -e NGC_API_SCHEME="$NGC_API_SCHEME" \
   -e NGC_AUTH_ENDPOINT="$NGC_AUTH_ENDPOINT" \
   -v "$CACHE_DIR:/opt/nim/.cache" \
   nim-docker-remote/nim/meta/llama-3.1-8b-instruct:1.1.0
```
It runs the llama container with GPU support, using environment variables for configuration, mounting a cache directory, and ensuring proper permissions.