Migrate Local Repositories

ARTIFACTORY: Hugging Face GA Repository Structure Migration

AuthorFullName__c
Yonatan Hen
articleNumber
000006001
ft:sourceType
Salesforce
FirstPublishedDate
2024-01-24T12:04:36Z
lastModifiedDate
2024-01-24
VersionNumber
5
To use the new repository structure for remote repositories:
1. For each model, download manually all the files of the model to the local disk using the JFrog Platform WebUI or REST API. To do so through the UI, go to the Artifacts page, right-click the file name, and select Download from the drop-down menu.

User-added image

2. Make sure that all the downloaded files for the same model are stored under the same folder.

✏️Important
Do not download the .jfrog_huggingface_model_info.json file for each model: it is an internal file for Artifactory usage.

3. Export your system variables using the following command:

✏️Note: 
Make sure to replace the placeholders in bold with your own IP address, Port, repository name, and Artifactory identity token
export HF_HUB_ETAG_TIMEOUT=1500000000
export HF_ENDPOINT=http://<ip>:<port>/artifactory/api/huggingfaceml/<repo-name>
export HF_TOKEN= <generated_artifactory_token>

For example:
export HF_HUB_ETAG_TIMEOUT=1500000000
export HF_ENDPOINT=http://<ip>:<port>/artifactory/api/huggingfaceml/<repo-name>
export HF_TOKEN= <generated_artifactory_token>

4. Deploy the models back to the local repository using the following command:

✏️Note: 
Make sure to replace the placeholders in bold with your own model source folder name, model name, and unique identifier.
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
    folder_path="<MODEL_SOURCE_FOLDER_NAME>", # Enter the name of the folder on your local machine where your model resides.
    repo_id="<MODEL_NAME>", # Enter the name of the model, following the Hugging Face repository naming structure 'organization/name'. (models--$<MODEL_NAME>)
    revision="<MODEL_UUID>", # Enter a unique identifier for your model. We recommend using a git commit revision ID. (snapshots/$<REVISION_NAME>/...files)   
    repo_type="model"
)

For example:
from huggingface_hub import HfApi
api = HfApi()
api.upload_folder(
    folder_path="vit-base-patch16-224", # Enter the name of the folder on your local machine where your model resides.folder to upload location on the FS
    repo_id="nsi319/legalpegasus", # Enter the name of the model, following the Hugging Face repository naming structure 'organization/name'. (models--$<MODEL_NAME>)
    revision="54ef2872d33bbff28eb09544bdecbf6699f5b0b8", # Enter a unique identifier for your model. We recommend using a Git commit revision ID. (snapshots/$<REVISION_NAME>/...files)    
    repo_type="model"
)

5. Delete all of the files from the old layout in the local repository. These are all the files that are stored outside of the timestamp folder in the artifact tree in the JFrog Platform WebUI.

User-added image