CODEWITHSUNDEEP
nginxtimeoutgitlab
Jan Ecker
Jan Ecker

Reputation: 65

Gitlab timeouts / slow on initial page loads

I am running Gitlab on Debian using the package from the Repository. Most of the time Gitlab is running very fast, but after longer idle times Gitlab is very slow or even times out (error 502). One time I also had a timeout on a remote git access (could not reproduce the issue - timeout on the internal API).

In my setup the the Debian machine is behind another nginx proxy which also serves some other services just fine. I did the gitlab-cli checks and everything seems fine.

In the error log of my reverse proxy I only see connection timeouts:

[error] 8643#0: *4139 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.1.1.10, server: gitlab.mydomain.tld, request: "GET / HTTP/1.1", upstream: "http://{SERVER-IP}:80/", host: "gitlab.mydomain.tld"

I can see some errors in my unicorn_stderr.log

E, [2016-03-30T19:40:20.183991 #783] ERROR -- : worker=1 PID:16798 timeout (61s > 60s), killing
E, [2016-03-30T19:40:20.194969 #783] ERROR -- : reaped #<Process::Status: pid 16798 SIGKILL (signal 9)> worker=1
I, [2016-03-30T19:40:20.197554 #16871]  INFO -- : worker=1 spawned pid=16871
I, [2016-03-30T19:40:20.197909 #16871]  INFO -- : worker=1 ready
E, [2016-03-30T20:08:42.911429 #783] ERROR -- : worker=0 PID:16866 timeout (61s > 60s), killing
E, [2016-03-30T20:08:43.191151 #783] ERROR -- : reaped #<Process::Status: pid 16866 SIGKILL (signal 9)> worker=0
I, [2016-03-30T20:08:43.758363 #18728]  INFO -- : worker=0 spawned pid=18728
I, [2016-03-30T20:08:44.108244 #18728]  INFO -- : worker=0 ready

What I am a bit curious about is the fact that there are no errors in the log of the nginx delivered with gitlab.

Some more system information:

#sudo gitlab-rake gitlab:env:info

System information
System:         Debian 8.3
Current User:   git
Using RVM:      no
Ruby Version:   2.1.8p440
Gem Version:    2.5.1
Bundler Version:1.10.6
Rake Version:   10.5.0
Sidekiq Version:4.0.1

GitLab information
Version:        8.5.0
Revision:       a513e09
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://gitlab.mydomain.tld
HTTP Clone URL: http://gitlab.mydomain.tld/some-group/some-project.git
SSH Clone URL:  [email protected]:some-group/some-project.git
Using LDAP:     no
Using Omniauth: no

GitLab Shell
Version:        2.6.10
Repositories:   /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git:            /opt/gitlab/embedded/bin/git

Edit:

My nginx config on the "external" reverse proxy looks like this:

server {
        listen  443;
        ssl     on;
        server_name gitlab.mydomain.tld;

        access_log  /var/log/nginx/gitlab.mydomain.tld.access.log;
        error_log   /var/log/nginx/gitlab.mydomain.tld.error.log;

        ssl_certificate         /etc/nginx/ssl/gitlab.mydomain.tld_unified.crt;
        ssl_certificate_key     /etc/nginx/ssl/mydomain.tld.key;

        location / {
                proxy_pass http://gitlab:80;
                proxy_redirect default;
                proxy_set_header        Host    $http_host;
                proxy_set_header        X_FORWARDED_PROTO "https";
                satisfy any;
        }
}

Edit2:

I took the suggested answer into account and also considered this source: https://github.com/gitlabhq/gitlabhq/blob/master/doc/install/requirements.md

I assigned 2GB RAM to the VM now, and also added one additional unicorn worker.

Edit3:

The problem seems to be solved by adding more memory and using 3 unicorn workers.

Upvotes: 4

Views: 3930

Answers (2)

Vasyl Korytov
Vasyl Korytov

Reputation: 66

FYI: I had to enlarge my small GitLab installation to have 4GB RAM not to throw OOM errors

Now I think, I'd better go with gogs or other alternative.

Upvotes: 0

Marco van Neerbos
Marco van Neerbos

Reputation: 536

Jan,

I have a similar setup although our box is dedicated to GITlab. Without knowing the specs of your server (GITLAB likes memory) and the load on that box I would suggest the following diagnostics:

  1. Does your upstream nginx use identical parameters as the gitlab nginx configuration? They have tweaked a number of things including timeouts.
  2. What kind of request result in time outs? Some operations (like generating diffs) can take some time to render.
  3. If you run the requests via SSH do you also experience time outs?
  4. Have you checked global logs in /var/log?

Upvotes: 1

Related Questions