How to troubleshoot Docker problems

Subject 

Comparing correct Docker REST API calls can show where the problem lies.

Affected Versions

4.0.2 – Latest

Description

When troubleshooting Docker errors it is best to compare what is supposed to happen against what is currently happening. You can eliminate a great deal of noise by simulating each step the client takes, and examining what HTTP return code is sent by Artifactory.

In general, these return codes indicate where to look next:
403 and 401 – Authentication issue, examine authorization settings
404 – Resource not found, check repository for resource
400 – Incorrect request, use curl to get reason phrase
50X – Server problem, examine artifactory.log

Resolution

Using curl
The libcurl terminal application can make basic REST API commands such as GET or PUT options. The Artifactory request logs display the exact API commands the Docker client uses.

You can use the examples below, or substitute the one that is failing in the Artifactory request.log file. Doing so should yield further information on the problem, and possible next steps.

Basic usage (with #comments):

curl -uadmin #Artifactory-username -vvv #verbosy -k #ignore-insecure-SSL -XGET #GET-request http://localhost:8081/artifactory/api/system/ping
 

Curl uses GET requests by default. Other commands like "PUT" require the –data field, along with the (Usually JSON) data payload. An example PUT command:

curl -uadmin:password -XPUT -H"Content-type: Application/json" –data '{"name":"test-group"}' http://localhost:8081/artifactory/api/security/groups/test-group

docker login docker.art.local
These are the backend steps that occur when a user tries to authenticate against Artifactory:
docker login docker.art.local

"docker" is the name of the virtual repository
*.art.local is the web address used by a reverse proxy configured for Subdomain Docker repositories

1. The Docker client checks the existence of Artifactory via an Anonymous GET request:

[$ARTIFACTORY_HOME/logs/request.log]
20180523095407|0|REQUEST|127.0.0.1|non_authenticated_user|GET|/api/docker/docker/v2/|HTTP/1.0|401|0

The 401 HTTP code is expected if Artifactory does not allow Anonymous access to the repository in question. If a 200 is seen by the Docker client credentials are not used.

2. The Docker client gets an access token:

20180523095417|10042|REQUEST|127.0.0.1|admin|GET|/api/docker/docker/v2/token|HTTP/1.0|200|0

This can be replicated via the following curl command:

curl -u<USERNAME> http://art.local:8081/artifactory/api/docker/docker/v2/token

Which returns a JSON with the token value:

{"token":"AKCp5Z3WYHpAyga9aYGuG4ESxLtC3Y1MAXUmfUemaKmrYvSpsesD4nsX2ezymTjMJLWowZ5LA","expires_in":3600}
 
docker pull docker.art.local/nginx
When a user attempts to pull an image, several steps should occur in sequence:
docker pull docker.art.local/nginx
1. Docker checks for the manifest.json of the file:
20180523095420|3182|REQUEST|127.0.0.1|admin|GET|/api/docker/docker/v2/nginx/manifests/latest|HTTP/1.0|200|948
2. The client uses the information in the manifest file to pull the image layers that make up the image:
20180523095420|41|REQUEST|127.0.0.1|admin|GET|/api/docker/docker/v2/nginx/blobs/sha256:ae513a47849c895a155ddfb868d6ba247f60240ec8495482eca74c4a2c13a881|HTTP/1.0|200|6028
20180523095423|2169|REQUEST|127.0.0.1|admin|GET|/api/docker/docker/v2/nginx/blobs/sha256:4a99993b863683bef1c776732e14d2372f6ed52b48e94783f4a1b58af289db07|HTTP/1.0|200|201
This completes the pull. If a layer is missing, the container's manifest should be inspected, and all layer requests should return a 200 OK.

docker push docker.art.local/ubuntu
When a user is ready to push an image, several checks occur before the file is uploaded:
docker push docker.art.local/ubuntu

1. The Docker client will check for the existence of the layers it needs to push:

20180523114900|65|REQUEST|127.0.0.1|admin|HEAD|/api/docker/docker/v2/ubuntu/blobs/sha256:471db38bcfbf0f5bac78012b9d458dfd37309d5cbb99d4e95310321a60a0cfdf|HTTP/1.0|404|0
A 404 not found is expected at this stage if the layer does not exist on Artifactory yet.

2. The client runs a POST command to submit the blob metadata:

20180523114900|22|REQUEST|127.0.0.1|admin|POST|/api/docker/docker/v2/ubuntu/blobs/uploads/|HTTP/1.0|202|0
3. The client will PATCH the rest of the blob metadata to Artifactory:
20180523114900|60|REQUEST|127.0.0.1|admin|PATCH|/api/docker/docker/v2/ubuntu/blobs/uploads/5582de7a-3127-48d9-bcb5-d61734f14e36|HTTP/1.0|202|0
4. Finally the client sends the layer binary:
20180523114900|122|REQUEST|127.0.0.1|admin|PUT|/api/docker/docker/v2/ubuntu/blobs/uploads/5582de7a-3127-48d9-bcb5-d61734f14e36|HTTP/1.0|201|0
5. Artifactory moves the binaries from a temporary _uploads folder to the Docker image's main location at the base directory of the repository:
 

Upload