How to solve the "upstream prematurely closed connection" Nginx / Apache2 error
Subject For long-running downloads, especially through a reverse proxy and using a client like Docker, adjusting a timeout value can resolve an error around unexpected timeouts.
Affected VersionsArtifactory 6.X and 7.X
Problem Description
This problem typically happens when downloading large files. These downloads naturally take, in this case, it's always more than a minute. The other component of this problem is having a reverse proxy or Load Balancer in front of the Artifactory application. The clients that have the most trouble with this issue are ones that download files in chunks, such as Docker.
If you've hit this issue, the Docker client will show you an error when pulling a large layer. You would not find any errors in the Artifactory logs, the only issue would be logged by Nginx:
2021/03/18 15:39:12 [error] 11196#11196: *526 upstream prematurely closed connection while reading upstream, client: 10.[...], server: ~(?<repo>.+)\.artifactory.com, request: "GET /v2/docker-local/blobs/sha256:d[...]e HTTP/1.1", upstream: "http://<Artifactory-IP>:8081/artifactory/api/docker/docker-local/v2/sha256:d[...]e", host: "docker-local.artifactory.com"
This error means the upstream, in this case, Artifactory, closed the connection.
Problem Resolution
Artifactory's Apache Tomcat has a hidden timeout setting. In some circumstances, such as a long-running Docker Pull, this timeout is reached. When this happens, Tomcat closes the connection too soon. No Artifactory-side errors get printed when this happens.
By default, the Tomcat "connectionTimeout" parameter is set to 60 seconds. To solve most Docker Pull problems, increase the value to 600 seconds, or 10 minutes:
[6.X - Directly update the tomcat server.xml] <Connector port="8081" sendReasonPhrase="true" relaxedPathChars='[]' relaxedQueryChars='[]' maxThreads="200" connectionTimeout="600000"/> [7.X - Update the system.yaml] artifactory: tomcat: connector: maxThreads: 200 extraConfig: 'connectionTimeout="600000"'