How to debug common errors related to Docker repositories in Artifactory running on K8s
Video Transcript:
Hello. My name is Paul Pan from JFrog Support. On today’s short the video I’m going to demonstrate to you How to debug common docker errors. Let’s jump in with an example. I’m going to try login into my docker registry and it felt with the v2 request. Okay. So the first technique I’m going to do is trying to run the v2, a curl command to hit this v2 location without authentication. As you can see, I got a full one from Artifactory. This is expected, but I’m also expected a realm in the return header. Okay? I’m not seeing that. So something is wrong with Artifactory handling this request. Let’s check why.
I go to the lock folder of Artifactory, and I’m going to tell the Artifactory request log. And I’m going to grep for the v2 request. And I’m going to try my login again. Okay. And I can immediately see that this is going to the v2 end point with a 401, but I’m not expecting it to hit directly at the v2 with the sub domain method that I configured. To demonstrate here is an example of how the sub domain method reverse proxy configuration is different from the others. So, when it hit the reverse proxy, I’m expecting it to rewrite the v2 request to Artifactory/api/docker prefix, with the repository name I’m passing. I’m not seeing it in the request log. So something must be wrong with my reverse proxy settings. Let’s take a look at what my reverse proxy settings is. Okay. As you can see, I am using a reverse proxy settings with sub domain method, so things looks good, except my server name here does not match what my server name was provided in the docker login command, so let’s try to modify it. I’m modifying my configuration. And let’s try to restart the nginx.
Okay. Now let’s restart it. Let’s do a docker login again. And now it is successful. Here, as you can see comparing to the direct access method, the things I need to pay attention to the reverse proxy sub domain is that in the server name I need to match in my docker host exactly, it was able to get to the repository name and do the rewrite of the v2 request correctly. Some additional settings in the reverse proxy that we need to pay attention to is the X-Jfrog-Override-Base-Url. This URL is going to be returned by Artifactory. And that’s how we get the realm URL in the return. Note that this URL has the port, which is going to be a grabbed from the nginx settings here, the host and the server port. In terms of more complicated settings, such as in Kubernetes environment it’s common to see that your client is hitting a load balancer first, before you hit the reverse proxy.
Example of that, I demonstrate the docker client hit the reverse proxy directly. So he was able to get the correct server ports from just the request. However, if you have a load balancer in front of the reverse proxy and you are terminating SSL at the load balancer, chances are when the requests hit from load balancer hit nginx, the port has already be changed. In the default settings, we’re relying on the settings to automatically get the server port from the request. But if you have a load balancer in front, you might want to hard-code the several port to be exactly the port that your load balancer is listed not.
Otherwise, the first need to request is going to be successful. But when you actually do the login, it will fail. Some additional technique I often use to debug in a Kubernetes environment is the port forward command. A port forward will allow you to forward a specific port or service directly to a local host. When debugging network issues, especially Docker on Kubernetes, Docker with Artifactory on Kubernetes, it is quite hard to isolate each network component. So the port forward would allow you to bypass certain network components all together to test your settings or test if simply the request was able to reach Artifactory or not.
Okay. That concludes my session for the day. If you have any questions and comments, feel free to leave below, and until next time.