How to solve the “upstream prematurely closed connection” Nginx / Apache2 error

Patrick Russell
2021-12-28 22:21

How to solve the "upstream prematurely closed connection" Nginx / Apache2 error

Subject 

For long-running downloads, especially through a reverse proxy and using a client like Docker, adjusting a timeout value can resolve an error around unexpected timeouts.

Affected Versions

Artifactory 6.X and 7.X

Problem Description

This problem typically happens when downloading large files. These downloads naturally take, in this case it's always more than a minute. The other component to this problem is having a reverse proxy or Load Balancer in front of the Artifactory application. The clients that have the most trouble with this issue are ones that download files in chunks, such as Docker.

If you've hit this issue, the Docker client will show you an error when pulling a large layer. You would not find any errors in the Artifactory logs, the only issue would be logged by Nginx:

2021/03/18 15:39:12 [error] 11196#11196: *526 upstream prematurely closed connection while reading upstream, client: 10.[…], server: ~(?<repo>.+).artifactory.com, request: "GET /v2/docker-local/blobs/sha256:d[…]e HTTP/1.1", upstream: "http://<Artifactory-IP>:8081/artifactory/api/docker/docker-local/v2/sha256:d[…]e", host: "docker-local.artifactory.com"

This error means the upstream, in this case Artifactory, closed the connection.

Problem Resolution

Artifactory's Apache Tomcat has a hidden timeout setting. In some circumstances, such as a long-running Docker Pull, this timeout is reached. When this happens, Tomcat closes the connection too soon. No Artifactory-side errors get printed when this happens.

By default the Tomcat "connectionTimeout" parameter is set to 60 seconds. To solve most Docker Pull problems, increase the value to 600 seconds, or 10 minutes:

[6.X – Directly update the tomcat server.xml]
<Connector port="8081" sendReasonPhrase="true" relaxedPathChars='[]' relaxedQueryChars='[]' maxThreads="200" connectionTimeout="600000"/>

[7.X – Update the system.yaml]
artifactory:
    tomcat:
        connector:
            maxThreads: 200
            extraConfig: 'connectionTimeout="600000"'