ARTIFACTORY: How to debug ‘unknown blob’ when pulling from a local docker repository

Affected Versions:

7.x 

Description:

When pulling a docker image from a local Artifactory repository or a virtual that only contains local repositories, you may get the error ‘unknown blob’: 

docker pull <artifactory>:8082/docker-local/library/ubuntu:20.04
20.04: Pulling from docker-local/library/ubuntu
8e5c1b329fe3: Pulling fs layer 
error pulling image configuration: download failed after attempts=1: unknown blob

Usually, this error means that the docker manifest.json references a layer that is not present in the local repository. We can look at the manifest.json (not the list):

{
    "schemaVersion": 2,
    "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
    "config": {
        "mediaType": "application/vnd.docker.container.image.v1+json",
        "size": 1463,
        "digest": "sha256:1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52"
    },
    "layers": [{
        "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
        "size": 28565998,
        "digest": "sha256:8e5c1b329fe39c318c0d49821b339fb94a215c5dc0a2898c8030b5a4d091bcba"
    }]
}

And we see that there are two referenced blobs in this manifest:
Sha256:8e5c1b329fe39c318c0d49821b339fb94a215c5dc0a2898c8030b5a4d091bcba
Sha256:1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52

We can check the logs or the UI to see which layer is missing by searching for 404s when pulling the image:

2022-04-28T22:36:52.271Z|6895fa07e993d5c6|52.9.243.19|admin|GET|/api/docker/docker-local/v2/library/ubuntu/blobs/sha256:1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52|404|-1|157|5|docker/20.10.14 go/go1.16.15 git-commit/87a90dc kernel/5.10.104-linuxkit os/linux arch/amd64 UpstreamClient(Docker-Client/20.10.14 \(darwin\))

In this case, it is 1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52 that is missing. 

Resolution:

The easiest way to fix this is by putting the layer back into this folder. We can do a checksum search https://www.jfrog.com/confluence/display/JFROG/Artifactory+REST+API#ArtifactoryRESTAPI-ChecksumSearch
for 1a437e363abfa47bfe4b3f5906b7444d12346102d944ebddd537e47a62fc6f52

Then if found, we can copy it into where it is supposed to be - the folder where the other layers exist. In my example, this would be under docker-local/library/ubuntu/sha256__7b3e30… note that this is not the ‘20.04’ folder. Note that the file name should be in docker layer format (sha256__<sha256 here>), and it has an attached property that is equal to sha256:<sha256 value>. 

If it does not exist in your Artifactory, then the image will have to be re-pushed into Artifactory from another source. If it is a public image, this will be as simple as re-caching it via a remote repository, then copying it into your local. Otherwise, you will need to locate the private source of the image and push it back into Artifactory.

Some causes:

Clean up scripts based off last downloaded timestamps can cause this to happen; it is not advisable to use these kinds of scripts on a docker repository unless it cleans up the entire image rather than some layers. 

Promotion:

Please be sure to use the docker specific promotion api for promotion
https://www.jfrog.com/confluence/display/JFROG/Docker+Registry#DockerRegistry-PromotingDockerImages