Python packages and model hubs like Hugging Face are commonly used in ML application and model development. To resolve dependencies from a public external repository (via proxy) or to store your own binaries produced during development, set up the appropriate repositories in Artifactory.
Create the following in Artifactory:
1. Remote PyPi Repository: to proxy all PyPi packages that will be needed from pypi.org
Set Up Remote PyPI Repositories
2. Virtual PyPi Repository: to aggregate requests for remote PyPi packages and any locally hosted or proprietary PyPi packages in a local PyPi repository in Artifactory
NOTE: Although this tutorial will not require the creation of a Local PyPi Repository, it is best practice to reference the Virtual PyPi Repo when configuring pip.
3. Local Hugging Face Repository: to store the custom ML Model version created by the SageMaker training job
Set Up Local Hugging Face Repositories
4. Remote Docker Repository: to proxy requests for Docker images needed from Docker Hub
Remote Docker Repositories
5. Local Docker Repository: to store a custom Docker image to be used for training
Local Docker Repositories
6. Virtual Docker Repository: to aggregate requests for both the Remote and Local Docker Repositories
Virtual Docker Repositories
7. Scoped Identity Token: to provide limited and scoped permission to access Artifactory from the SageMaker environment
Generating Scoped Tokens
Create the following in Artifactory:
1. Remote PyPi Repository: to proxy all PyPi packages that will be needed from pypi.org
Set Up Remote PyPI Repositories
2. Virtual PyPi Repository: to aggregate requests for remote PyPi packages and any locally hosted or proprietary PyPi packages in a local PyPi repository in Artifactory
NOTE: Although this tutorial will not require the creation of a Local PyPi Repository, it is best practice to reference the Virtual PyPi Repo when configuring pip.
3. Local Hugging Face Repository: to store the custom ML Model version created by the SageMaker training job
Set Up Local Hugging Face Repositories
4. Remote Docker Repository: to proxy requests for Docker images needed from Docker Hub
Remote Docker Repositories
5. Local Docker Repository: to store a custom Docker image to be used for training
Local Docker Repositories
6. Virtual Docker Repository: to aggregate requests for both the Remote and Local Docker Repositories
Virtual Docker Repositories
7. Scoped Identity Token: to provide limited and scoped permission to access Artifactory from the SageMaker environment
Generating Scoped Tokens