GitLab-EE not accessible after update from 14.1.3 to 14.2
Summary
After upgrading GitLab-EE Omnibus installation from 14.1.3 to 14.2, the server responds with "502 Whoops, GitLab is taking too much time to respond.".
No errors were shown during installation with apt install, gitlab-ctl reconfigure or gitlab-ctl restart.
uname -a
Linux hw-git-mirror 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64 GNU/LinuxSteps to reproduce
Current installed version gitlab-ee 14.1.3-ee.0 Update Omnibus installation to 14.2 Try connect to GitLab's Web-Interface after 5 minutes
DB migration has finished
sudo gitlab-rails runner -e production 'puts Gitlab::BackgroundMigration.remaining'
0What is the current bug behavior?
GitLab permanently shows HTTP error "502 Whoops, GitLab is taking too much time to respond."
What is the expected correct behavior?
GitLab should show the login page after DB migration and other startup procedures.
Relevant logs and/or screenshots
administrator@hw-git-mirror:~$ sudo gitlab-ctl reconfigure
...
Running handlers:
Running handlers complete
Chef Infra Client finished, 1/790 resources updated in 38 seconds
gitlab Reconfigured!
administrator@hw-git-mirror:~$ sudo gitlab-ctl restart
ok: run: alertmanager: (pid 29723) 0s
ok: run: gitaly: (pid 29736) 0s
ok: run: gitlab-exporter: (pid 29753) 0s
ok: run: gitlab-workhorse: (pid 29755) 1s
ok: run: grafana: (pid 29769) 0s
ok: run: logrotate: (pid 29777) 1s
ok: run: nginx: (pid 29788) 0s
ok: run: node-exporter: (pid 29794) 0s
ok: run: postgres-exporter: (pid 29800) 0s
ok: run: postgresql: (pid 29808) 0s
ok: run: prometheus: (pid 29848) 1s
ok: run: puma: (pid 29912) 0s
ok: run: redis: (pid 29917) 0s
ok: run: redis-exporter: (pid 29924) 0s
ok: run: sidekiq: (pid 30012) 1s
administrator@hw-git-mirror:~$ sudo gitlab-ctl tail
==> /var/log/gitlab/node-exporter/current <==
2021-08-23_15:37:12.03612 level=info ts=2021-08-23T15:37:11.985Z caller=node_exporter.go:113 collector=thermal_zone
2021-08-23_15:37:12.03612 level=info ts=2021-08-23T15:37:11.985Z caller=node_exporter.go:113 collector=time
2021-08-23_15:37:12.03612 level=info ts=2021-08-23T15:37:11.985Z caller=node_exporter.go:113 collector=timex
2021-08-23_15:37:12.03612 level=info ts=2021-08-23T15:37:11.986Z caller=node_exporter.go:113 collector=udp_queues
2021-08-23_15:37:12.03613 level=info ts=2021-08-23T15:37:11.986Z caller=node_exporter.go:113 collector=uname
2021-08-23_15:37:12.03613 level=info ts=2021-08-23T15:37:11.986Z caller=node_exporter.go:113 collector=vmstat
2021-08-23_15:37:12.03614 level=info ts=2021-08-23T15:37:11.986Z caller=node_exporter.go:113 collector=xfs
2021-08-23_15:37:12.03614 level=info ts=2021-08-23T15:37:11.986Z caller=node_exporter.go:113 collector=zfs
2021-08-23_15:37:12.03615 level=info ts=2021-08-23T15:37:11.987Z caller=node_exporter.go:195 msg="Listening on" address=localhost:9100
2021-08-23_15:37:12.03615 level=info ts=2021-08-23T15:37:12.022Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
==> /var/log/gitlab/node-exporter/state <==
==> /var/log/gitlab/unicorn/current <==
2020-06-02_08:09:18.69855 Terminated
2020-06-02_08:09:18.70576 forwarding TERM to unicorn master 23215
2020-06-02_08:09:18.72477 Waiting for unicorn to die...
2020-06-02_08:09:19.72710 Waiting for unicorn to die...
2020-06-02_08:09:20.72967 Waiting for unicorn to die...
2020-06-02_08:09:21.73221 Waiting for unicorn to die...
2020-06-02_08:09:22.73584 Waiting for unicorn to die...
2020-06-02_08:09:23.74489 Forcibly killing unicorn
2020-06-02_08:09:23.74493 forwarding CONT to unicorn master 23215
2020-06-02_08:09:23.75604 wrapper for unicorn master 23215 exiting
==> /var/log/gitlab/unicorn/unicorn_stdout.log <==
==> /var/log/gitlab/unicorn/state <==
==> /var/log/gitlab/unicorn/unicorn_stderr.log <==
==> /var/log/gitlab/sidekiq/current <==
{"severity":"INFO","time":"2021-08-23T15:38:15.207Z","message":"Cron Jobs - add job with name: network_policy_metrics_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.210Z","message":"Cron Jobs - add job with name: iterations_update_status_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.213Z","message":"Cron Jobs - add job with name: iterations_generator_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.216Z","message":"Cron Jobs - add job with name: vulnerability_statistics_schedule_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.230Z","message":"Cron Jobs - add job with name: vulnerability_historical_statistics_deletion_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.233Z","message":"Cron Jobs - add job with name: security_create_orchestration_policy_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.236Z","message":"Cron Jobs - add job with name: security_orchestration_policy_rule_schedule_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.239Z","message":"Cron Jobs - add job with name: app_sec_dast_profile_schedule_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.349Z","message":"Cron Jobs - add job with name: update_all_mirrors_worker"}
{"severity":"INFO","time":"2021-08-23T15:38:15.354Z","message":"Cron Jobs - add job with name: geo_sidekiq_cron_config_worker"}
==> /var/log/gitlab/sidekiq/state <==
==> /var/log/gitlab/gitaly/current <==
{"level":"info","msg":"7fff486a8000-7fff486ab000 r--p 00000000 00:00 0 [vvar]","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","29741","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-08-23T17:37:39.281Z"}
{"level":"info","msg":"7fff486ab000-7fff486ad000 r-xp 00000000 00:00 0 [vdso]","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","29741","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-08-23T17:37:39.281Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","29741","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-08-23T17:37:39.281Z"}
{"level":"info","msg":"","supervisor.args":["bundle","exec","bin/ruby-cd","/var/opt/gitlab/gitaly","/opt/gitlab/embedded/service/gitaly-ruby/bin/gitaly-ruby","29741","/var/opt/gitlab/gitaly/internal_sockets/ruby.0"],"supervisor.name":"gitaly-ruby.0","time":"2021-08-23T17:37:39.281Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/ruby.0 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.0: connect: connection refused\". Reconnecting...","pid":29741,"system":"system","time":"2021-08-23T17:37:40.056Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: connection refused\". Reconnecting...","pid":29741,"system":"system","time":"2021-08-23T17:37:40.078Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/ruby.0 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.0: connect: connection refused\". Reconnecting...","pid":29741,"system":"system","time":"2021-08-23T17:37:55.057Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: connection refused\". Reconnecting...","pid":29741,"system":"system","time":"2021-08-23T17:37:55.079Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.0 /var/opt/gitlab/gitaly/internal_sockets/ruby.0 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.0: connect: connection refused\". Reconnecting...","pid":29741,"system":"system","time":"2021-08-23T17:38:10.058Z"}
{"level":"warning","msg":"[core] grpc: addrConn.createTransport failed to connect to {/var/opt/gitlab/gitaly/internal_sockets/ruby.1 /var/opt/gitlab/gitaly/internal_sockets/ruby.1 \u003cnil\u003e 0 \u003cnil\u003e}. Err: connection error: desc = \"transport: Error while dialing dial unix /var/opt/gitlab/gitaly/internal_sockets/ruby.1: connect: connection refused\". Reconnecting...","pid":29741,"system":"system","time":"2021-08-23T17:38:10.081Z"}
==> /var/log/gitlab/gitaly/gitlab-shell.log <==
==> /var/log/gitlab/gitaly/gitaly_hooks.log <==
==> /var/log/gitlab/gitaly/state <==
==> /var/log/gitlab/gitaly/gitaly_ruby_json.log <==
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:05Z","grpc.time_ms":0.332,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6082,"correlation_id":"0fe236021cee1aa89460a88cafb99a7d","time":"2021-08-23T15:14:05.588Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:05Z","grpc.time_ms":0.274,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6081,"correlation_id":"ca7565a8c81f85592ed1e96ec5218d95","time":"2021-08-23T15:14:05.590Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:20Z","grpc.time_ms":0.27,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6081,"correlation_id":"4673b0c6bcc945c5c72ab260abfbcfd3","time":"2021-08-23T15:14:20.795Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:20Z","grpc.time_ms":0.252,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6082,"correlation_id":"302e5b573d1501a95995b96d494f9291","time":"2021-08-23T15:14:20.815Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:35Z","grpc.time_ms":6.849,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6081,"correlation_id":"676806b64e925ab3a0a02dbe7f40db6e","time":"2021-08-23T15:14:35.837Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:35Z","grpc.time_ms":3.338,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6082,"correlation_id":"1bdd375b6b56b1f71a41c4c7348a3ccd","time":"2021-08-23T15:14:35.838Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:50Z","grpc.time_ms":9.765,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6082,"correlation_id":"0961d8b29e936ad0786462b30899229a","time":"2021-08-23T15:14:50.868Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:50Z","grpc.time_ms":0.41,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6081,"correlation_id":"5c284446709fcd3a8cac9a0d61feaf64","time":"2021-08-23T15:14:50.873Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:53Z","grpc.time_ms":0.362,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6081,"correlation_id":"4384d5e4a6213acfc5b5fe4e5411739c","time":"2021-08-23T15:14:53.483Z"}
{"type":"gitaly-ruby","grpc.start_time":"2021-08-23T15:14:53Z","grpc.time_ms":0.375,"grpc.code":"OK","grpc.method":"Check","grpc.service":"grpc.health.v1.Health","pid":6082,"correlation_id":"3f5b13d10e6594c7bc93c383e71209ff","time":"2021-08-23T15:14:53.510Z"}
==> /var/log/gitlab/nginx/current <==
==> /var/log/gitlab/nginx/gitlab_access.log <==
172.16.7.31 - - [23/Aug/2021:17:37:53 +0200] "GET / HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:37:53 +0200] "GET /favicon.ico HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:37:57 +0200] "GET / HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:37:57 +0200] "GET /favicon.ico HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:38:00 +0200] "GET / HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:38:00 +0200] "GET /favicon.ico HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:38:05 +0200] "GET / HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:38:05 +0200] "GET /favicon.ico HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:38:08 +0200] "GET / HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
172.16.7.31 - - [23/Aug/2021:17:38:08 +0200] "GET /favicon.ico HTTP/2.0" 502 2940 "https://hw-git-mirror/" "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0" -
==> /var/log/gitlab/nginx/access.log <==
==> /var/log/gitlab/nginx/state <==
==> /var/log/gitlab/nginx/error.log <==
==> /var/log/gitlab/nginx/gitlab_error.log <==
==> /var/log/gitlab/alertmanager/current <==
2021-08-23_15:37:08.61579 level=info ts=2021-08-23T15:37:08.615Z caller=main.go:221 msg="Starting Alertmanager" version="(version=0.22.2, branch=master, revision=)"
2021-08-23_15:37:08.61582 level=info ts=2021-08-23T15:37:08.615Z caller=main.go:222 build_context="(go=go1.16.4, user=GitLab-Omnibus, date=)"
2021-08-23_15:37:08.61862 level=info ts=2021-08-23T15:37:08.618Z caller=cluster.go:184 component=cluster msg="setting advertise address explicitly" addr=172.16.7.3 port=9094
2021-08-23_15:37:08.62005 level=info ts=2021-08-23T15:37:08.619Z caller=cluster.go:671 component=cluster msg="Waiting for gossip to settle..." interval=2s
2021-08-23_15:37:08.71584 level=info ts=2021-08-23T15:37:08.715Z caller=coordinator.go:113 component=configuration msg="Loading configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml
2021-08-23_15:37:08.71616 level=info ts=2021-08-23T15:37:08.716Z caller=coordinator.go:126 component=configuration msg="Completed loading of configuration file" file=/var/opt/gitlab/alertmanager/alertmanager.yml
2021-08-23_15:37:08.72166 level=info ts=2021-08-23T15:37:08.721Z caller=main.go:514 msg=Listening address=localhost:9093
2021-08-23_15:37:08.72311 level=info ts=2021-08-23T15:37:08.722Z caller=tls_config.go:191 msg="TLS is disabled." http2=false
2021-08-23_15:37:10.62064 level=info ts=2021-08-23T15:37:10.620Z caller=cluster.go:696 component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.00048957s
2021-08-23_15:37:18.62278 level=info ts=2021-08-23T15:37:18.622Z caller=cluster.go:688 component=cluster msg="gossip settled; proceeding" elapsed=10.002637366s
==> /var/log/gitlab/alertmanager/state <==
==> /var/log/gitlab/grafana/current <==
2021-08-23_15:37:14.09678 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096760+02:00","queryType":"arrow"}
2021-08-23_15:37:14.09679 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096777+02:00","queryType":"annotations"}
2021-08-23_15:37:14.09681 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096789+02:00","queryType":"table_static"}
2021-08-23_15:37:14.09681 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096803+02:00","queryType":"random_walk_with_error"}
2021-08-23_15:37:14.09683 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096816+02:00","queryType":"server_error_500"}
2021-08-23_15:37:14.09684 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096830+02:00","queryType":"logs"}
2021-08-23_15:37:14.09685 {"@level":"debug","@message":"datasource: registering query type handler","@timestamp":"2021-08-23T17:37:14.096841+02:00","queryType":"node_graph"}
2021-08-23_15:37:14.09687 {"@level":"debug","@message":"datasource: registering query type fallback handler","@timestamp":"2021-08-23T17:37:14.096850+02:00"}
2021-08-23_15:37:14.16091 t=2021-08-23T17:37:14+0200 lvl=eror msg="Failed to read plugin provisioning files from directory" logger=provisioning.plugins path=/var/opt/gitlab/grafana/provisioning/plugins error="open /var/opt/gitlab/grafana/provisioning/plugins: no such file or directory"
2021-08-23_15:37:14.30396 t=2021-08-23T17:37:14+0200 lvl=info msg="HTTP Server Listen" logger=http.server address=127.0.0.1:3000 protocol=http subUrl=/-/grafana socket=
==> /var/log/gitlab/grafana/state <==
==> /var/log/gitlab/redis/current <==
2021-08-23_15:37:16.09130 29917:C 23 Aug 2021 17:37:16.091 # Configuration loaded
2021-08-23_15:37:16.14318 29917:M 23 Aug 2021 17:37:16.142 * Running mode=standalone, port=0.
2021-08-23_15:37:16.14321 29917:M 23 Aug 2021 17:37:16.142 # Server initialized
2021-08-23_15:37:16.14322 29917:M 23 Aug 2021 17:37:16.142 # WARNING overcommit_memory is set to 0! Background save may fail under low memory condition. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
2021-08-23_15:37:16.14323 29917:M 23 Aug 2021 17:37:16.142 # WARNING you have Transparent Huge Pages (THP) support enabled in your kernel. This will create latency and memory usage issues with Redis. To fix this issue run the command 'echo madvise > /sys/kernel/mm/transparent_hugepage/enabled' as root, and add it to your /etc/rc.local in order to retain the setting after a reboot. Redis must be restarted after THP is disabled (set to 'madvise' or 'never').
2021-08-23_15:37:16.16754 29917:M 23 Aug 2021 17:37:16.162 * Loading RDB produced by version 6.0.14
2021-08-23_15:37:16.16757 29917:M 23 Aug 2021 17:37:16.162 * RDB age 1 seconds
2021-08-23_15:37:16.16758 29917:M 23 Aug 2021 17:37:16.162 * RDB memory usage when created 8.73 Mb
2021-08-23_15:37:16.18912 29917:M 23 Aug 2021 17:37:16.189 * DB loaded from disk: 0.027 seconds
2021-08-23_15:37:16.18916 29917:M 23 Aug 2021 17:37:16.189 * The server is now ready to accept connections at /var/opt/gitlab/redis/redis.socket
==> /var/log/gitlab/redis/state <==
==> /var/log/gitlab/gitlab-workhorse/current <==
{"correlation_id":"01FDSTJKFSCRATE8RDWWWDZ0Z1","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-08-23T17:38:00+02:00","uri":"/favicon.ico"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FDSTJKFSCRATE8RDWWWDZ0Z1","duration_ms":0,"host":"hw-git-mirror","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://hw-git-mirror/","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-08-23T17:38:00+02:00","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0","written_bytes":2940}
{"correlation_id":"01FDSTJQYF7CBQNRZBPXDNP88K","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-08-23T17:38:05+02:00","uri":"/"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FDSTJQYF7CBQNRZBPXDNP88K","duration_ms":0,"host":"hw-git-mirror","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://hw-git-mirror/","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-08-23T17:38:05+02:00","ttfb_ms":0,"uri":"/","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0","written_bytes":2940}
{"correlation_id":"01FDSTJQZ4782JGFP6F2PA4G2Q","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-08-23T17:38:05+02:00","uri":"/favicon.ico"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FDSTJQZ4782JGFP6F2PA4G2Q","duration_ms":0,"host":"hw-git-mirror","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://hw-git-mirror/","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-08-23T17:38:05+02:00","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0","written_bytes":2940}
{"correlation_id":"01FDSTJV946TJXYGZKZ8RJ85EQ","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-08-23T17:38:08+02:00","uri":"/"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FDSTJV946TJXYGZKZ8RJ85EQ","duration_ms":0,"host":"hw-git-mirror","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://hw-git-mirror/","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-08-23T17:38:08+02:00","ttfb_ms":0,"uri":"/","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0","written_bytes":2940}
{"correlation_id":"01FDSTJV9YV3H6MMC4CF6J4P3Z","duration_ms":0,"error":"badgateway: failed to receive response: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused","level":"error","method":"GET","msg":"","time":"2021-08-23T17:38:08+02:00","uri":"/favicon.ico"}
{"content_type":"text/html; charset=utf-8","correlation_id":"01FDSTJV9YV3H6MMC4CF6J4P3Z","duration_ms":0,"host":"hw-git-mirror","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"https://hw-git-mirror/","remote_addr":"127.0.0.1:0","remote_ip":"127.0.0.1","route":"","status":502,"system":"http","time":"2021-08-23T17:38:08+02:00","ttfb_ms":0,"uri":"/favicon.ico","user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0","written_bytes":2940}
==> /var/log/gitlab/gitlab-workhorse/state <==
==> /var/log/gitlab/postgresql/current <==
2021-08-23_15:37:12.55026 FATAL: terminating connection due to administrator command
2021-08-23_15:37:12.55026 FATAL: terminating connection due to administrator command
2021-08-23_15:37:12.55026 FATAL: terminating connection due to administrator command
2021-08-23_15:37:12.68075 LOG: background worker "logical replication launcher" (PID 20881) exited with exit code 1
2021-08-23_15:37:12.70566 LOG: shutting down
2021-08-23_15:37:12.96892 LOG: database system is shut down
2021-08-23_15:37:13.51916 LOG: starting PostgreSQL 12.6 on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2021-08-23_15:37:13.53747 LOG: listening on Unix socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"
2021-08-23_15:37:13.90010 LOG: database system was shut down at 2021-08-23 15:37:12 GMT
2021-08-23_15:37:14.12424 LOG: database system is ready to accept connections
==> /var/log/gitlab/postgresql/state <==
==> /var/log/gitlab/gitlab-shell/gitlab-shell.log <==
==> /var/log/gitlab/redis-exporter/current <==
2021-08-23_14:11:42.62272 time="2021-08-23T16:11:42+02:00" level=info msg="Redis Metrics Exporter 1.17.0 build date: sha1: Go: go1.16.4 GOOS: linux GOARCH: amd64"
2021-08-23_14:11:42.62581 time="2021-08-23T16:11:42+02:00" level=info msg="Providing metrics at localhost:9121/metrics"
2021-08-23_14:29:47.12469 time="2021-08-23T16:29:47+02:00" level=info msg="Redis Metrics Exporter 1.17.0 build date: sha1: Go: go1.16.4 GOOS: linux GOARCH: amd64"
2021-08-23_14:29:47.12473 time="2021-08-23T16:29:47+02:00" level=info msg="Providing metrics at localhost:9121/metrics"
2021-08-23_14:48:22.44612 time="2021-08-23T16:48:22+02:00" level=info msg="Redis Metrics Exporter 1.17.0 build date: sha1: Go: go1.16.4 GOOS: linux GOARCH: amd64"
2021-08-23_14:48:22.44633 time="2021-08-23T16:48:22+02:00" level=info msg="Providing metrics at localhost:9121/metrics"
2021-08-23_15:14:56.55655 time="2021-08-23T17:14:56+02:00" level=info msg="Redis Metrics Exporter 1.17.0 build date: sha1: Go: go1.16.4 GOOS: linux GOARCH: amd64"
2021-08-23_15:14:56.58821 time="2021-08-23T17:14:56+02:00" level=info msg="Providing metrics at localhost:9121/metrics"
2021-08-23_15:37:16.74939 time="2021-08-23T17:37:16+02:00" level=info msg="Redis Metrics Exporter 1.17.0 build date: sha1: Go: go1.16.4 GOOS: linux GOARCH: amd64"
2021-08-23_15:37:16.77868 time="2021-08-23T17:37:16+02:00" level=info msg="Providing metrics at localhost:9121/metrics"
==> /var/log/gitlab/redis-exporter/state <==
==> /var/log/gitlab/puma/current <==
2021-08-23_15:37:14.23414 Received TERM from runit, sending to process group (-PID)
2021-08-23_15:37:20.66459 {"timestamp":"2021-08-23T15:37:20.664Z","pid":29912,"message":"Puma starting in cluster mode..."}
2021-08-23_15:37:20.66768 {"timestamp":"2021-08-23T15:37:20.667Z","pid":29912,"message":"* Puma version: 5.3.2 (ruby 2.7.2-p137) (\"Sweetnighter\")"}
2021-08-23_15:37:20.66775 {"timestamp":"2021-08-23T15:37:20.667Z","pid":29912,"message":"* Min threads: 4"}
2021-08-23_15:37:20.66781 {"timestamp":"2021-08-23T15:37:20.667Z","pid":29912,"message":"* Max threads: 4"}
2021-08-23_15:37:20.66786 {"timestamp":"2021-08-23T15:37:20.667Z","pid":29912,"message":"* Environment: production"}
2021-08-23_15:37:20.66791 {"timestamp":"2021-08-23T15:37:20.667Z","pid":29912,"message":"* Master PID: 29912"}
2021-08-23_15:37:20.66797 {"timestamp":"2021-08-23T15:37:20.667Z","pid":29912,"message":"* Workers: 2"}
2021-08-23_15:37:20.66804 {"timestamp":"2021-08-23T15:37:20.668Z","pid":29912,"message":"* Restarts: (✔) hot (✖) phased"}
2021-08-23_15:37:20.66808 {"timestamp":"2021-08-23T15:37:20.668Z","pid":29912,"message":"* Preloading application"}
==> /var/log/gitlab/puma/puma_stderr.log <==
7ff1ec756000-7ff1ec774000 r-xp 00001000 08:01 3408641 /lib/x86_64-linux-gnu/ld-2.28.so
7ff1ec774000-7ff1ec77c000 r--p 0001f000 08:01 3408641 /lib/x86_64-linux-gnu/ld-2.28.so
7ff1ec77c000-7ff1ec77d000 r--p 00026000 08:01 3408641 /lib/x86_64-linux-gnu/ld-2.28.so
7ff1ec77d000-7ff1ec77e000 rw-p 00027000 08:01 3408641 /lib/x86_64-linux-gnu/ld-2.28.so
7ff1ec77e000-7ff1ec77f000 rw-p 00000000 00:00 0
7ffcb608e000-7ffcb6a8d000 rw-p 00000000 00:00 0 [stack]
7ffcb6bb9000-7ffcb6bbc000 r--p 00000000 00:00 0 [vvar]
7ffcb6bbc000-7ffcb6bbe000 r-xp 00000000 00:00 0 [vdso]
==> /var/log/gitlab/puma/state <==
==> /var/log/gitlab/puma/puma_stdout.log <==
{"timestamp":"2021-08-23T15:35:23.343Z","pid":19967,"message":"PumaWorkerKiller: Consuming 2685.6953125 mb with master and 2 workers."}
{"timestamp":"2021-08-23T15:35:43.344Z","pid":19967,"message":"PumaWorkerKiller: Consuming 2686.41015625 mb with master and 2 workers."}
{"timestamp":"2021-08-23T15:35:53.168Z","pid":19967,"message":"- Worker 0 (PID: 28844) booted in 0.4s, phase: 0"}
{"timestamp":"2021-08-23T15:36:03.345Z","pid":19967,"message":"PumaWorkerKiller: Consuming 2682.4140625 mb with master and 2 workers."}
{"timestamp":"2021-08-23T15:36:23.346Z","pid":19967,"message":"PumaWorkerKiller: Consuming 2686.81640625 mb with master and 2 workers."}
{"timestamp":"2021-08-23T15:36:43.346Z","pid":19967,"message":"PumaWorkerKiller: Consuming 2687.4765625 mb with master and 2 workers."}
{"timestamp":"2021-08-23T15:37:03.347Z","pid":19967,"message":"PumaWorkerKiller: Consuming 2687.6953125 mb with master and 2 workers."}
{"timestamp":"2021-08-23T15:37:14.357Z","pid":19967,"message":"=== puma shutdown: 2021-08-23 17:37:14 +0200 ==="}
{"timestamp":"2021-08-23T15:37:14.357Z","pid":19967,"message":"- Goodbye!"}
{"timestamp":"2021-08-23T15:37:14.357Z","pid":19967,"message":"- Gracefully shutting down workers..."}
==> /var/log/gitlab/gitlab-exporter/current <==
2021-08-23_15:37:54.66397 ::1 - - [23/Aug/2021:17:37:54 CEST] "GET /sidekiq HTTP/1.1" 200 102564
2021-08-23_15:37:54.66406 - -> /sidekiq
2021-08-23_15:37:58.98586 ::1 - - [23/Aug/2021:17:37:58 CEST] "GET /ruby HTTP/1.1" 200 993
2021-08-23_15:37:58.98593 - -> /ruby
2021-08-23_15:38:07.81719 ::1 - - [23/Aug/2021:17:38:07 CEST] "GET /database HTTP/1.1" 200 1492
2021-08-23_15:38:07.81727 - -> /database
2021-08-23_15:38:09.63034 ::1 - - [23/Aug/2021:17:38:09 CEST] "GET /sidekiq HTTP/1.1" 200 102564
2021-08-23_15:38:09.63042 - -> /sidekiq
2021-08-23_15:38:14.01237 ::1 - - [23/Aug/2021:17:38:13 CEST] "GET /ruby HTTP/1.1" 200 993
2021-08-23_15:38:14.01246 - -> /ruby
==> /var/log/gitlab/gitlab-exporter/state <==
==> /var/log/gitlab/prometheus/current <==
2021-08-23_15:37:24.57352 level=info ts=2021-08-23T15:37:24.573Z caller=main.go:802 msg="TSDB started"
2021-08-23_15:37:24.57356 level=info ts=2021-08-23T15:37:24.573Z caller=main.go:928 msg="Loading configuration file" filename=/var/opt/gitlab/prometheus/prometheus.yml
2021-08-23_15:37:24.58411 level=error ts=2021-08-23T15:37:24.575Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-08-23_15:37:24.58419 level=error ts=2021-08-23T15:37:24.584Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-08-23_15:37:24.58436 level=error ts=2021-08-23T15:37:24.584Z caller=manager.go:314 component="discovery manager scrape" msg="Cannot create service discovery" err="unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" type=kubernetes
2021-08-23_15:37:24.61959 level=info ts=2021-08-23T15:37:24.619Z caller=main.go:959 msg="Completed loading of configuration file" filename=/var/opt/gitlab/prometheus/prometheus.yml totalDuration=45.948265ms remote_storage=2.739µs web_handler=1.143µs query_engine=1.404µs scrape=110.879µs scrape_sd=9.14309ms notify=32.136µs notify_sd=24.829µs rules=34.820842ms
2021-08-23_15:37:24.61962 level=info ts=2021-08-23T15:37:24.619Z caller=main.go:751 msg="Server is ready to receive web requests."
2021-08-23_15:37:29.62034 level=error ts=2021-08-23T15:37:29.620Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-nodes
2021-08-23_15:37:29.62042 level=error ts=2021-08-23T15:37:29.620Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-pods
2021-08-23_15:37:29.62065 level=error ts=2021-08-23T15:37:29.620Z caller=manager.go:188 component="scrape manager" msg="error creating new scrape pool" err="error creating HTTP client: unable to load specified CA cert /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: open /var/run/secrets/kubernetes.io/serviceaccount/ca.crt: no such file or directory" scrape_pool=kubernetes-cadvisor
==> /var/log/gitlab/prometheus/state <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-01-03-08-52-03.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-08-23-15-10-50.log <==
-- indexes(:design_user_mentions)
-> 0.0039s
-- execute("SET statement_timeout TO 0")
-> 0.0003s
-- remove_index(:design_user_mentions, {:algorithm=>:concurrently, :name=>"design_user_mentions_on_design_id_and_note_id_index"})
-> 0.0030s
-- execute("RESET statement_timeout")
-> 0.0005s
== 20210818055357 AddUniqueCommitDesignUserMentionIndexes: migrated (0.0414s) =
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-04-27-16-29-36.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-07-06-21-01-35.log <==
==> /var/log/gitlab/gitlab-rails/grpc.log <==
==> /var/log/gitlab/gitlab-rails/auth.log <==
{"severity":"INFO","time":"2021-08-23T14:49:05.840Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T14:49:05.840Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:10:15.421Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:10:53.240Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:15:49.419Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:15:49.419Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:22:54.470Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:25:08.400Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:38:08.037Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
{"severity":"INFO","time":"2021-08-23T15:38:08.037Z","correlation_id":null,"gitlab_throttle_user_allowlist":[]}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-07-06-21-14-39.log <==
==> /var/log/gitlab/gitlab-rails/service_measurement.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-07-02-16-07-12.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-08-19-09-14-08.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-04-06-10-50-03.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-11-02-14-31-43.log <==
==> /var/log/gitlab/gitlab-rails/application.log <==
2021-08-23T15:33:49.668Z: ActiveRecord connection established
2021-08-23T15:33:57.412Z: ActiveRecord connection established
2021-08-23T15:34:05.234Z: ActiveRecord connection established
2021-08-23T15:34:10.970Z: ActiveRecord connection established
2021-08-23T15:34:25.005Z: ActiveRecord connection established
2021-08-23T15:34:31.471Z: ActiveRecord connection established
2021-08-23T15:34:49.801Z: ActiveRecord connection established
2021-08-23T15:35:53.167Z: ActiveRecord connection established
2021-08-23T15:38:07.614Z: Checking state of dynamic postgres partitions
2021-08-23T15:38:07.620Z: Checking state of dynamic postgres partitions
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-07-16-18-56-53.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-01-05-17-31-15.log <==
==> /var/log/gitlab/gitlab-rails/git_json.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-07-07-09-19-43.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-02-13-10-54-17.log <==
==> /var/log/gitlab/gitlab-rails/production.log <==
Creating scope :not_checksummed. Overwriting existing method MergeRequestDiff.not_checksummed.
Creating scope :without_statuses. Overwriting existing method CommitStatus.without_statuses.
Creating scope :without_statuses. Overwriting existing method Ci::Pipeline.without_statuses.
Creating scope :in_pipelines. Overwriting existing method Ci::Build.in_pipelines.
Creating scope :of_projects. Overwriting existing method MergeRequest.of_projects.
Creating scope :join_project. Overwriting existing method MergeRequest.join_project.
Creating scope :references_project. Overwriting existing method MergeRequest.references_project.
Creating scope :system. Overwriting existing method Note.system.
Creating scope :closed. Overwriting existing method Iteration.closed.
Creating scope :within_timeframe. Overwriting existing method Iteration.within_timeframe.
==> /var/log/gitlab/gitlab-rails/api_json.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2021-03-09-17-57-56.log <==
==> /var/log/gitlab/gitlab-rails/sidekiq_client.log <==
==> /var/log/gitlab/gitlab-rails/graphql_json.log <==
==> /var/log/gitlab/gitlab-rails/exceptions_json.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-05-04-20-10-01.log <==
==> /var/log/gitlab/gitlab-rails/production_json.log <==
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:34:48.453Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"redis_calls":1,"redis_duration_s":0.001211,"redis_read_bytes":203,"redis_write_bytes":74,"redis_cache_calls":1,"redis_cache_duration_s":0.001211,"redis_cache_read_bytes":203,"redis_cache_write_bytes":74,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.049862,"mem_objects":4201,"mem_bytes":482416,"mem_mallocs":3396,"mem_total_bytes":650456,"pid":28247,"correlation_id":"d4d6ac73-62fb-4d4b-ba94-adb0e810544b","db_duration_s":0.00331,"view_duration_s":0.05125,"duration_s":0.06178}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:35:03.375Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.02143,"mem_objects":1635,"mem_bytes":139920,"mem_mallocs":1135,"mem_total_bytes":205320,"pid":28247,"correlation_id":"70428e50-a247-40d2-8bbe-e9606e81754d","db_duration_s":0.0,"view_duration_s":0.00107,"duration_s":0.01396}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:35:18.477Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"redis_calls":1,"redis_duration_s":0.001301,"redis_read_bytes":203,"redis_write_bytes":74,"redis_cache_calls":1,"redis_cache_duration_s":0.001301,"redis_cache_read_bytes":203,"redis_cache_write_bytes":74,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.051636,"mem_objects":4190,"mem_bytes":455024,"mem_mallocs":3387,"mem_total_bytes":622624,"pid":28298,"correlation_id":"934bf807-9321-4a5d-9482-362ef3e70cc1","db_duration_s":0.00341,"view_duration_s":0.06213,"duration_s":0.0847}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:35:33.369Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.014578,"mem_objects":1635,"mem_bytes":139944,"mem_mallocs":1135,"mem_total_bytes":205344,"pid":28298,"correlation_id":"d2796f91-f3b4-4124-ad26-570287ba8e58","db_duration_s":0.0,"view_duration_s":0.00078,"duration_s":0.00863}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:35:48.371Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.014419,"mem_objects":1633,"mem_bytes":156280,"mem_mallocs":1135,"mem_total_bytes":221600,"pid":28247,"correlation_id":"b508b857-c01c-4a92-bcda-d456ecd35dd9","db_duration_s":0.0,"view_duration_s":0.00068,"duration_s":0.0115}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:36:03.424Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"redis_calls":1,"redis_duration_s":0.002253,"redis_read_bytes":203,"redis_write_bytes":74,"redis_cache_calls":1,"redis_cache_duration_s":0.002253,"redis_cache_read_bytes":203,"redis_cache_write_bytes":74,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.052216,"mem_objects":4190,"mem_bytes":455024,"mem_mallocs":3387,"mem_total_bytes":622624,"pid":28844,"correlation_id":"3f37a8a6-9208-463b-b4c2-f68fb538edce","db_duration_s":0.0031,"view_duration_s":0.02747,"duration_s":0.03812}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:36:18.369Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.014426,"mem_objects":1604,"mem_bytes":137480,"mem_mallocs":1116,"mem_total_bytes":201640,"pid":28247,"correlation_id":"29ec3d9e-2ff7-4317-92c7-6a2ef9791d35","db_duration_s":0.0,"view_duration_s":0.00073,"duration_s":0.00835}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:36:33.386Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.014307,"mem_objects":1635,"mem_bytes":139944,"mem_mallocs":1135,"mem_total_bytes":205344,"pid":28844,"correlation_id":"e8a9e856-71e8-40fb-bd54-dfc4a29da559","db_duration_s":0.0,"view_duration_s":0.00083,"duration_s":0.02325}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:36:48.389Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.014953,"mem_objects":1599,"mem_bytes":137320,"mem_mallocs":1114,"mem_total_bytes":201280,"pid":28247,"correlation_id":"f2366d68-5c4c-4480-9a61-5187faba0008","db_duration_s":0.0,"view_duration_s":0.00087,"duration_s":0.02948}
{"method":"GET","path":"/-/metrics","format":"html","controller":"MetricsController","action":"index","status":200,"time":"2021-08-23T15:37:03.375Z","params":[],"remote_ip":null,"user_id":null,"username":null,"ua":null,"db_count":0,"db_write_count":0,"db_cached_count":0,"cpu_s":0.020032,"mem_objects":1633,"mem_bytes":139896,"mem_mallocs":1134,"mem_total_bytes":205216,"pid":28844,"correlation_id":"056f8d9c-2103-430d-84a4-db4300a7f1d9","db_duration_s":0.0,"view_duration_s":0.00088,"duration_s":0.01308}
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-06-02-10-09-43.log <==
==> /var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-2020-08-25-21-09-16.log <==
...Output of checks
administrator@hw-git-mirror:~$ sudo gitlab-ctl status
run: alertmanager: (pid 21291) 905s; run: log: (pid 595) 11259s
run: gitaly: (pid 21299) 904s; run: log: (pid 627) 11258s
run: gitlab-exporter: (pid 21317) 903s; run: log: (pid 601) 11259s
run: gitlab-workhorse: (pid 21319) 903s; run: log: (pid 614) 11259s
run: grafana: (pid 21335) 902s; run: log: (pid 596) 11259s
run: logrotate: (pid 21344) 902s; run: log: (pid 600) 11259s
run: nginx: (pid 21350) 902s; run: log: (pid 594) 11259s
run: node-exporter: (pid 21357) 901s; run: log: (pid 597) 11259s
run: postgres-exporter: (pid 21365) 901s; run: log: (pid 599) 11259s
run: postgresql: (pid 21374) 900s; run: log: (pid 598) 11259s
run: prometheus: (pid 21376) 900s; run: log: (pid 604) 11259s
run: puma: (pid 21474) 898s; run: log: (pid 626) 11258s
run: redis: (pid 21479) 898s; run: log: (pid 621) 11259s
run: redis-exporter: (pid 21486) 898s; run: log: (pid 605) 11259s
run: sidekiq: (pid 21494) 895s; run: log: (pid 593) 11259sadministrator@hw-git-mirror:~$ sudo gitlab-ctl check-config
administrator@hw-git-mirror:~$ Edited by Carsten Stelling