ARTIFACTORY: How to debug docker configuration issues

Scott Mosher
2022-06-23 09:15

Docker debugging and understanding what to expect from the incoming requests

In this article, we will not go over the Docker configuration steps outlined already in our wiki at https://www.jfrog.com/confluence/display/JFROG/Docker+Registry, but will help understand what the configuration achieves, how it works with Artifactory and how to debug if we run into issues with the configuration.

Each Docker method (Repository Path, sub-domain, and Port) is used to format requests in a way Artifactory expects.  It does not ultimately change how Artifactory handles incoming Docker requests.  We will go through the expected requests that we can parse for each request type.

Docker login:

We will always see 401s with the Docker client during the initial authentication request and this is expected.  The Docker client sends credentials once the 401 is returned.  
|non_authenticated_user|GET|/api/docker/v2/|401|
|user|GET|/api/docker/null/v2/token|200|  (the null in this request can also be the repository name we are logging into, but in this request, we are getting a token for the Artifactory registry itself)
|user|GET|/api/docker/v2/|200|

We can further debug to gather a bit more information by mimicking the Docker login via curl (this is repository path specific request, but can be updated for other methods) e.g.

curl -uuser http://PLATFORM_URL/v2/DOCKER_REPO/token -v

 

Pulling Docker images:

For Docker pulls, we don’t expect the same 401s.  We now have the token, so we should see the user authenticated within the request.  We now expect HEAD and GET requests.
|user|HEAD|/api/docker/docker-remote/v2/some-image/manifests/some-tag|200|

We then start to see the manifest pull.
|user|GET|/api/docker/docker-remote/v2/some-image/manifests/sha256:12321321adfasdf1f6a2a14b56891f10b6c87b7d7b846253f95934dd6932|200|

Once we get the manifest/s, we can then pull the actual blobs.
|user|GET|/api/docker/docker-remote/v2/some-image/blobs/sha256:eaaf2ec477ce7fcb2bc1f6a2a14b1f10b6c87b7d77b846253f95934dd6932|200|
|user|GET|/api/docker/docker-remote/v2/some-image/blobs/sha256:1f10b6c87b7d79222265c132a5231af7c8584a907e2afc6df0634b78ba98f|200|

Again, we can further test out this pull with a curl request.  We will be using the token returned from the previous login request we ran.

curl -H "Authorization: Bearer <TOKEN>" -vk
https://PLATFORM_URL/v2/docker/hello-world/manifests/latest

(in this example docker is a virtual repository that contains a Docker remote repository pointing to dockerhub)

Pushing Docker images

For Docker pushes, these requests are not as straightforward.  Again, all these requests should be authenticated and will initially be HEAD requests.
|user|HEAD|/api/docker/docker-local/v2/path/some-image/blobs/sha256:be248e90f762e2a43bb52249552e74eceb8e629f1392c664468bf946ea12472c|404|
|user|HEAD|/api/docker/docker-local/v2/path/some-image/blobs/sha256:73dd2a7a91633255c1c8687f3a368b651f6cdc87637e1aa3e6a67ac7868a296b|404|

We then see the POST command uploading to the temporary uploads directory.
|user|POST|/api/docker/docker/v2/path/some-images/blobs/uploads/|202|
|user|POST|/api/docker/docker/v2/path/some-images/blobs/uploads/|202|

The then should see following PATCH and PUT requests for the blobs and the last request would be the manifest/tag.
|user|PATCH|/api/docker/docker-local/v2/path/some-image/blobs/uploads/b3526fec-5803-4733-a1ac-e52e7d09173b|202|
|user|PATCH|/api/docker/docker-local/v2/path/some-image/blobs/uploads/7953cdb7-9a59-4b7e-bab9-fcbde155e972|202|
|user|PUT|/api/docker/docker-local/v2/path/some-image/blobs/uploads/b3526fec-5803-4733-a1ac-e52e7d09173b|201|
|user|PUT|/api/docker/docker-local/v2/path/some-image/blobs/uploads/9ce1ffc5-bf15-4eae-b4dd-2bd030aa8dc6|201|
|user|PUT|/api/docker/docker-local/v2/path/some-image/manifests/latest|201|

 

Understanding proxy configuration and how to debug

We will take a generic nginx reverse proxy configuration as this or Apache would be used when setting up the sub-domain Docker method.  The necessary configuration that is required is to set the servername and rewrite.

server_name ~(?<repo>.+).PLATFORM_URL PLATFORM_URL;
rewrite ^/(v1|v2)/(.*) /artifactory/api/docker/$repo/$1/$2; 

This takes the repository provided in the subdomain to construct the request as Artifactory expects.  We can also look to the headers we set if we continue to see issues.

proxy_set_header    X-Forwarded-Port  $server_port;
proxy_set_header    X-Forwarded-Proto $http_x_forwarded_proto;
proxy_set_header    Host              $http_host;

The POST commands above will take the header set within the request and push the PATCH requests to this location.  If we see the POST commands hit Artifactory as expected, but the PATCH requests are failing, this usually is a result of an invalid header.

Putting this all together

Taking the above expected requests along with understanding the purpose of the reverse proxy configuration allows for us to further debug and pinpoint possible places where the issue originates.  Even without a reverse proxy, we still can further troubleshoot by just understanding what we expect in the logs.  

The first thing to confirm is the request structure.  We expect to see within the artifactory-request.log, that each request starts with the /api/docker/REPO_NAME.  If we don’t see this format, we can then point to the Docker request or reverse proxy configuration as the cause of the issue as Artifactory is receiving an incorrectly formed request.

A couple examples of bad requests and what cause them:

|user|POST|/api/docker//v2/path/some-images/blobs/uploads/|404|   

(the repository name is missing which could be caused by the reverse proxy not having the $repo variable set properly)

|user|GET|/v2|202|   

(caused by the v2 request from the Docker client not being rewritten to /api/docker/REPO_NAME/…)

|user|GET|/api/docker/some-image/|404|   

(Docker pull request does not have the repository name within the request)

Another couple of common causes for Docker issues within the Artifactory configuration would be the Docker method we are using does not match the selected Docker method under the Artifactory -> Http Settings tab on the Admin panel.  We also need to confirm the Default Deployment Repository is set within the virtual repository we are hitting.