openssl:1.1.1f causes "unable to get local issuer certificate"
TL/DR: Please upgrade the bundled openssl to 1.1.1g!
Summary
After upgrading to Gitlab 12.9.3-ce.0 (and afterwards also to 12.10.0-ce.0), Gitlab greeted everyone trying to login with the following error message:
(certificate verify failed (unable to get local issuer certificate)
Gitlab was no longer able to verify the certificate of the LDAP server, therefore effectively blocking all logins.
Oh oh, that's not good - fortunately we tried the upgrade in a test environment first.
A long troubleshooting session began - we could reproduce the issue with the bundled openssl binary. So somehow openssl seemed to handle things differently now than it used to do.
To confirm this, we spun up some openssl docker containers - and voila, only openssl:1.1.1f triggered this behaviour, not openssl:1.1.1e, and not openssl:1.1.1g. Unfortunately this is the version Gitlab-omnibus currently bundles.
Combing through the openssl issue tracker revealed #11456 (upgrading from 1.1.1.e and 1.1.1.f causes openvpn connections using ssl to fail) a.k.a OpenSSL "changed the validation result for self-signed certificates with invalid certificate extensions", although "they are syntactically correct and openssl itself can produce such certificates without any errors", apparently "because [OpenSSL] is a foot-shoot-gun".
And, in fact, our LDAP server certificate contained the following extension:
$ openssl x509 -in server.pem -text -noout
[...]
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:FALSE, pathlen:0
... which finally is the reason the certificate is rejected, and the login fails.
In the meantime, openssl reverted this change as it "is breaking existing installations".
As apparently you cannot downgrade the bundled openssl again (as dbf998fc mentions older versions cause failures in Bitbucket Cloud - what is going on with openssl?!), I ask you to consider upgrading to 1.1.1g (released on 21-Apr-2020), which fixed this bug, contains another security fix, and hopefully introduces no new breakages.
Steps to reproduce
- Use Gitlab >= 12.9.3-ce.0
- Have a LDAP server using a certificate with "Basic Constraints CA:false, pathlen:0", that violates the RFC 5280 although "openssl itself can produce such certificates without any errors"
- Notice the error "unable to get local issuer certificate" when trying to login
What is the current bug behavior?
Verification fails with certain certificates
What is the expected correct behavior?
OpenSSL verifies the certificates correctly
Relevant logs
==> unicorn/unicorn_stdout.log <==
{"severity":"ERROR","timestamp":"2020-04-23T09:51:27.054Z","pid":31891,"progname":"omniauth","message":"(ldapmain) Authentication failure! ldap_error: Net::LDAP::Error, SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"}
Details of package version
Gitlab 12.9.3-ce.0 and 12.10.0-ce.0
# /opt/gitlab/embedded/bin/openssl version
OpenSSL 1.1.1f 31 Mar 2020
Environment details
- Operating System: Ubuntu 16.04, Ubuntu 18.04, installed via gitlab-omnibus repository
- Installation Target:
- VM: *umCloud
- Installation Type:
- New Installation and Upgrades