CODEWITHSUNDEEP
wgetartifactorylarge-files
Tom
Tom

Reputation: 31

Artifactory large downloads failing

My continuing saga:

I have Artifactory Pro 5.8.4 running on a CentOS 7.4 system that uses Artifactory's built-in Nginx service as a reverse proxy. Most of the time, during download of a large object, the connection fails. Additionally, the last few kilobytes of data that are transferred in the failed connection appear to be garbage; as a result, the file, when resumed, has a section in the middle somewhere that is invalid. Nginx logs around the time of failure provide the following error message:

2018/01/22 23:26:32 [error] 884#0: *48048865 readv() failed (104: Connection reset by peer) while reading upstream, client: 12.2.178.130, server: ~(?.+).artifactory.common.mycoolco.com, request: "GET /controller/release/20171218-2119-2aa9f0e8c32bf3f08dd96a97127204d1f400f895fa2e67100a3830861c0becb0bfae28244c058510c8312e98378c0b77251b4bb7926ceae2d465ce2d533b88dc HTTP/1.1", upstream: "http://127.0.0.1:8081/artifactory/controller/release/20171218-2119-2aa9f0e8c32bf3f08dd96a97127204d1f400f895fa2e67100a3830861c0becb0bfae28244c058510c8312e98378c0b77251b4bb7926ceae2d465ce2d533b88dc", host: "artifactory.common.mycoolco.com"

The Artifactory logs usually, but not always, have an error that looks something like this:

2018-01-22 23:14:20,106 [http-nio-8081-exec-87] [WARN ] (o.a.r.ArtifactoryResponseBase:137) - Client Closed Request 499: java.net.SocketTimeoutException

Any further suggestions for diagnosis would be appreciated.

THIS DID NOT AFFECT ANY NOTICEABLE CHANGE: As a follow-through, I will be changing some settings for testing over this weekend (SUN, MAR. 18TH) during my maintenance window by editing the nginx.conf file (/etc/opt/rh/rh-nginx18/nginx/nginx.conf) to contain the settings "worker_processes 2" (was currently 1), upping worker connections from 1024 to 2048 and changing the keepalive_timeout from 65 to 130.

We have a single Artifactory instance hosted in AWS with no other proxies nor load balancers; just a single instance and a route 53 entry pointing to the private ip address. We are seeing a baseline 10%cpu usage with spikes to 100% every 15 minutes ...apparently from the from the java process. java -version: openjdk version "1.8.0_161" OpenJDK Runtime Environment (build 1.8.0_161-b14) OpenJDK 64-Bit Server VM (build 25.161-b14, mixed mode). now on artifactory.version: 5.8.4

> --2018-03-08 22:09:42--  https://artifactory.common.mycoolco.com/controller/pre-release/20180308-2216-023a0b8309af8889
Resolving artifactory.common.mycoolco.com (artifactory.common.mycoolco.com)... 55.111.66.99
Connecting to artifactory.common.mycoolco.com (artifactory.common.mycoolco.com)|55.111.66.99|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1241911191 (1.2G) [application/octet-stream]
Saving to: ‘20180308-2216-023a0b8309af8889’

20180308-2216-c3499ea023a0b8309  86%[=============================================>        ]   1.00G   568KB/s    in 1m 31s 

2018-03-08 22:30:14 (857 KB/s) - Connection closed at byte 1079774172. Retrying.

--2018-03-08 22:30:15--  (try: 2)  https://artifactory.common.mycoolco.com/controller-os/pre-release/20180308-2216-023a0b8309af8889
Connecting to artifactory.common.mycoolco.com (artifactory.common.mycoolco.com)|55.111.66.99|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1241911191 (1.2G), 162137019 (155M) remaining [application/octet-stream]
Saving to: ‘20180308-2216-023a0b8309af8889’

20180308-2216-c3499ea023a0b8309 100%[++++++++++++++++++++++++++++++++++++++++++++++=======>]   1.16G  1.08MB/s    in 0m 22s  

2018-03-08 22:32:38 (1.09 MB/s) - ‘20180308-2216-023a0b8309af8889’ saved [1241911191/1241911191]

top command showing java high cpu usage AWS cpu spikes

Upvotes: 3

Views: 7832

Answers (2)

sky67
sky67

Reputation: 71

The issue could be due to ​proxy_max_temp_file_size. The default value is 1GB. Check if proxy_max_temp_file_size 0 helps.

Upvotes: 7

Ariel
Ariel

Reputation: 3506

By the error message, it seems that the issue is the 'Socket Timeout (MS)' parameter on your remote repository (as you mentioned in your comment). In order to resolve the issue go to Artifactory UI --> Remote repositories --> Edit the repository that is having the issue --> Go to Advanced Tab --> Edit the 'Socket Timeout (MS)' param and extend it.

If this is not a remote repository but local repository on a remote instance, it probably an issue with the proxy in front of Artifactory.

Upvotes: 1

Related Questions