ARTIFACTORY: Hugging Face GA Repository Structure Migration

Description

Starting from Artifactoty 7.77.X, you can access Hugging Face GA in Self-hosted environments. The GA version includes an update to the Hugging Face repository structure, which might cause storage issues in the local repositories you created during the beta.

The layout change will make the stored file inaccessible to Hugging Face APIs and have the following side effects:

Remote repositories: the models will be downloaded again from Hugging Face to the Artifactory cache
Local repositories: the models are inaccessible and require a re-upload.

To learn more about the new features included in the GA version, see the release notes.


Local Repository Structure Changes Overview

The Hugging Face local repository before Artifactory 7.77.x is a beta version, with the following storage structure:

<local-repository>
    |
    —-models
        |
        —organization (optional)
            |
            —model_name
                    |
                    |—-main
                        |
                        —-model files

For example:

User-added image

The Hugging Face local repository after Artifactory 7.77.x is a GA with the structure of:

<local-repository>
    |
    —-models
        |
        —organization (optional)
            |
            —model_name
                    |—-main
                         |-----<UPLOAD_DATE>
                               |
                               |----------model files

For example:

User-added image

Migrate Remote Repositories

To use the new repository structure for remote repositories recache the files by downloading them again from Hugging Face.


Migrate Local Repositories

To use the new repository structure for remote repositories:
1. For each model, download manually all the files of the model to the local disk using the JFrog Platform WebUI or REST API. To do so through the UI, go to the Artifacts page, right-click the file name, and select Download from the drop-down menu.

User-added image

2. Make sure that all the downloaded files for the same model are stored under the same folder.

✏️Important
Do not download the .jfrog_huggingface_model_info.json file for each model: it is an internal file for Artifactory usage.

3. Export your system variables using the following command:

✏️Note:
Make sure to replace the placeholders in bold with your own IP address, Port, repository name, and Artifactory identity token

export HF_HUB_ETAG_TIMEOUT=1500000000
export HF_ENDPOINT=http://<ip>:<port>/artifactory/api/huggingfaceml/<repo-name>
export HF_TOKEN= <generated_artifactory_token>

For example:

export HF_HUB_ETAG_TIMEOUT=1500000000
export HF_ENDPOINT=http://<ip>:<port>/artifactory/api/huggingfaceml/<repo-name>
export HF_TOKEN= <generated_artifactory_token>

4. Deploy the models back to the local repository using the following command:

✏️Note:

Make sure to replace the placeholders in bold with your own model source folder name, model name, and unique identifier.

from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
    folder_path="<MODEL_SOURCE_FOLDER_NAME>", # Enter the name of the folder on your local machine where your model resides.
    repo_id="<MODEL_NAME>", # Enter the name of the model, following the Hugging Face repository naming structure 'organization/name'. (models--$<MODEL_NAME>)
    revision="<MODEL_UUID>", # Enter a unique identifier for your model. We recommend using a git commit revision ID. (snapshots/$<REVISION_NAME>/...files)   
    repo_type="model"
)

For example:

from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
    folder_path="vit-base-patch16-224", # Enter the name of the folder on your local machine where your model resides.folder to upload location on the FS
    repo_id="nsi319/legalpegasus", # Enter the name of the model, following the Hugging Face repository naming structure 'organization/name'. (models--$<MODEL_NAME>)
    revision="54ef2872d33bbff28eb09544bdecbf6699f5b0b8", # Enter a unique identifier for your model. We recommend using a Git commit revision ID. (snapshots/$<REVISION_NAME>/...files)    
    repo_type="model"
)

5. Delete all of the files from the old layout in the local repository. These are all the files that are stored outside of the timestamp folder in the artifact tree in the JFrog Platform WebUI.

User-added image