Skip to content

Webterminal behind Reverse Proxy with Let's Encrypt Cert gives "x509: certificate signed by unknown authority" error

Summary

We are trying to enable Webterminal for our Gitlab-CE instance. Direct access to the nodes with the gitlab-runners is working. The Webterminal connects and we can access the containers.

[...]
Terminal is connected, will time out in 30m0s...
Terminal disconnected
Job succeeded

The gitlab-runners are running in our Kubernetes cluster (which is otherwise independent of Gitlab), access to the runners (for now) is made available via Ingress L4 load balancing.

To make sure we can always access our runners, even in case of a node shutdown, we use our reverse proxy for load balancing and SSL termination. We have a valid wildcard Let's Encrypt certificate installed:

# curl -v https://gitlab-runner.ourcompany.com
[...]
* Server certificate:
*  subject: CN=ourcompany.com
*  start date: Mar 22 08:51:56 2019 GMT
*  expire date: Jun 20 08:51:56 2019 GMT
*  subjectAltName: host "gitlab-runner.ourcompany.com" matched cert's "*.ourcompany.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
[...]
< 
404 page not found
* Connection #0 to host gitlab-runner.ourcompany.com left intact

However, when we tell Gitlab to connect to the webterminal via the reverse proxy, it fails with the following entry in gitlab-workhorse logs:

2019-03-26_10:28:08.94238 time="2019-03-26T11:28:08+01:00" level=error msg=error correlation_id=vQ8Feb9qu76 error="x509: certificate signed by unknown authority" method=GET uri=/group/ci-tests/-/jobs/8868/terminal.ws

It seems the Let's Encrypt CA is not part of the cacert.pem file:

# openssl verify -CAfile /opt/gitlab/embedded/ssl/certs/cacert.pem our-lets-encrypt-cert.pem
CN = ourcompany.com
error 20 at 0 depth lookup: unable to get local issuer certificate
error our-lets-encrypt-cert.pem: verification failed

Following the steps in https://docs.gitlab.com/omnibus/settings/ssl.html#connecting-to-external-resources we added the Let's Encrypt CA chain:

# ls /opt/gitlab/embedded/ssl/certs
2e5ac55d.0  4f06f81d.0  cacert.pem  README
# openssl verify -CApath /opt/gitlab/embedded/ssl/certs/ -show_chain our-lets-encrypt-cert.pem
our-lets-encrypt-cert.pem: OK
Chain:
depth=0: CN = ourcompany.com (untrusted)
depth=1: C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
depth=2: O = Digital Signature Trust Co., CN = DST Root CA X3

Sadly, this does not change anything.

Are we missing something? We are not very familiar with the internal works of gitlab, so it might be a user error.

We tried several things:

  • adding the certificates directly to the cacert.pem file
  • setting the SSL_CERT_DIR env variable for workhorse to /etc/ssl/certs where all necessary CA certs are contained.
  • testing the connection with gitlab-rails console and Net::HTTP.get(URI(...)), no errors there.

Steps to reproduce

  1. Setup gitlab-runner with session_server enabled.
  2. Make sure it works with direct connecting gitlab-workhorse -> gitlab-runner
  3. Setup a reverse proxy with a valid Let's Encrypt certificate
  4. Copy the certificate chain to /etc/gitlab/trusted-certs and run gitlab-ctl reconfigure
  5. Update the runner config such that gitlab-workhorse connects to the reverse proxy for the webterminal

What is the current bug behavior?

It seems the added Let's Encrypt CA is not loaded und thus the certificate cannot be validated.

What is the expected correct behavior?

The presented certificate of the reverse proxy is validated against the provided Let's Encrypt CA chain and the connection can be established.

Relevant logs and/or screenshots

See summary.

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         Ubuntu 18.04
Current User:   git
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.16.6
Rake Version:   12.3.2
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.5
Go Version:     unknown


GitLab information
Version:        11.9.0
Revision:       a47124c7852
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.mycompany.com
HTTP Clone URL: https://gitlab.mycompany.com/some-group/some-project.git
SSH Clone URL:  ssh://git@gitlab.mycompany.com:1848/some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        8.7.1
Repository storage paths:
  • default: /projects/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)


(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)


(we will only investigate if the tests are passing)

Details of package version

Provide the package version installation details 
# dpkg-query -l "gitlab-*"
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                                            Version                      Architecture                 Description
+++-===============================================-============================-============================-===================================================================================================
ii  gitlab-ce                                       11.9.1-ce.0                  amd64                        GitLab Community Edition (including NGINX, Postgres, Redis)
un  gitlab-ee                                                                                     (no description available)

Environment details

  • Operating System: Ubuntu 18.04.1 LTS
  • Installation Target, remove incorrect values:
    • VM: KVM
  • Installation Type, remove incorrect values:
    • Updated: Always kept up-to-date
  • Is there any other software running on the machine: besides orchestration, no
  • Is this a single or multiple node installation? single
  • Resources
    • CPU: 4
    • Memory total: 6G

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
external_url 'https://gitlab.mycompany.com'
git_data_dirs({"default"=>{"path"=>"/projects/gitlab/git-data"}})
gitlab_rails['backup_archive_permissions'] = 511
gitlab_rails['backup_keep_time'] = 1209600
gitlab_rails['backup_path'] = "/storage/backup/gitlab"
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_database'] = "gitlab"
gitlab_rails['db_host'] = "xxx"
gitlab_rails['db_password'] = "xxx"
gitlab_rails['db_pool'] = 10
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['gitlab_default_can_create_group'] = false
gitlab_rails['gitlab_default_projects_features_builds'] = false
gitlab_rails['gitlab_default_projects_features_issues'] = true
gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
gitlab_rails['gitlab_default_projects_features_snippets'] = false
gitlab_rails['gitlab_default_projects_features_visibility_level'] = "private"
gitlab_rails['gitlab_default_projects_features_wiki'] = false
gitlab_rails['gitlab_email_from'] = "gitlab@mycompany.com"
gitlab_rails['gitlab_restricted_visibility_levels'] = ["public"]
gitlab_rails['gitlab_shell_ssh_port'] = 1234
gitlab_rails['gitlab_ssh_host'] = "gitlab.mycompany.com"
gitlab_rails['gitlab_username_changing_enabled'] = false
gitlab_rails['incoming_email_address'] = "gitlab+%{key}@service.mycompany.com"
gitlab_rails['incoming_email_email'] = "gitlab@service.mycompany.com"
gitlab_rails['incoming_email_enabled'] = true
gitlab_rails['incoming_email_host'] = "mail.mycompany.com"
gitlab_rails['incoming_email_mailbox_name'] = "inbox"
gitlab_rails['incoming_email_password'] = "xxx"
gitlab_rails['incoming_email_port'] = 143
gitlab_rails['incoming_email_ssl'] = false
gitlab_rails['incoming_email_start_tls'] = true
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = xxx
gitlab_rails['lfs_enabled'] = true
gitlab_rails['lfs_storage_path'] = "/projects/gitlab/git-lfs"
gitlab_rails['manage_backup_path'] = false
gitlab_rails['omniauth_enabled'] = false
gitlab_rails['time_zone'] = "xxx"
gitlab_rails['trusted_proxies'] = ["the proxy in question is contained in this list"]
sidekiq['concurrency'] = 5
postgresql['enable'] = false
nginx['listen_https'] = false
nginx['listen_port'] = 80
nginx['redirect_http_to_https'] = false
logging['svlogd_filter'] = "gzip"
logging['svlogd_num'] = 30
logging['svlogd_size'] = 209715200
logging['svlogd_timeout'] = 86400
prometheus_monitoring['enable'] = true
prometheus['enable'] = false
node_exporter['enable'] = false
redis_exporter['enable'] = true
redis_exporter['listen_address'] = "0.0.0.0:9121"
postgres_exporter['enable'] = false
gitlab_monitor['enable'] = true
gitlab_monitor['listen_address'] = "0.0.0.0"
gitlab_monitor['listen_port'] = 9168
pages_external_url 'http://pages.mycompany.com'
gitlab_pages['access_control'] = true
pages_nginx['redirect_http_to_https'] = false
gitaly['prometheus_listen_addr'] = "0.0.0.0:9175"

Possible fixes

Sadly, none.

Edited by Ben Prescott_