Support TLS in dedicated metrics exporters
What does this MR do and why?
For FIPS compliance, all metrics endpoints scraped by Prometheus need to support TLS. We provide two mechanisms to serve metrics from the Rails monolith:
-
Rails controller endpoint (
/-/metrics
- Puma only): This is covered by #353013 (closed) and related MRs. -
Dedicated server endpoint (
/metrics
- Puma and Sidekiq): This MR. This is currently a WEBrick that runs on a separate port. We need to configure it with a certificate and key to enable TLS. We are also working on a replacement system written in Go (gitlab-metrics-exporter) which isn't used yet, but I already added the config mapping, too.
WEBrick supports HTTPS out of the box. We merely need to set the proper SSL config keys. These are mapped from exporter settings in gitlab.yml
. See omnibus-gitlab!6155 (merged) for how it's done in Omnibus.
How to set up and validate locally
In gitlab.yml
, set the following keys:
monitoring:
web_exporter: # or sidekiq_exporter, it's the same implementation
enabled: true # should already be true in development
tls_enabled: true
tls_cert_path: /path/to/cert.pem
tls_key_path: /path/to/key.pem
You could use the test cert and PK from the fixtures folder: https://gitlab.com/gitlab-org/gitlab/blob/72e0329c8d86e804f9ea152b590a73ce591a003e/spec/fixtures/x509_certificate.crt
Then, run Puma (or Sidekiq), and curl
the configured address
and port
, e.g.:
$ curl -v https://localhost:8083/metrics >/dev/null
* Trying 127.0.0.1:8083...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Connected to localhost (127.0.0.1) port 8083 (#0)
* ALPN, offering http/1.1
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [122 bytes data]
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
{ [10 bytes data]
* TLSv1.3 (IN), TLS handshake, Certificate (11):
{ [1271 bytes data]
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
{ [520 bytes data]
* TLSv1.3 (IN), TLS handshake, Finished (20):
{ [52 bytes data]
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.3 (OUT), TLS handshake, Finished (20):
} [52 bytes data]
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=AU; CN=localhost
* start date: May 11 14:00:47 2022 GMT
* expire date: May 8 14:00:47 2032 GMT
* subjectAltName: host "localhost" matched cert's "localhost"
* issuer: C=AU; CN=example
* SSL certificate verify ok.
} [5 bytes data]
> GET /metrics HTTP/1.1
> Host: localhost:8083
> User-Agent: curl/7.79.1-DEV
> Accept: */*
>
{ [5 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
{ [265 bytes data]
* old SSL session ID is stale, removing
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4
< Vary: Accept-Encoding
< Server: WEBrick/1.6.1 (Ruby/2.7.5/2021-11-24)
< Date: Wed, 15 Jun 2022 10:48:53 GMT
< Content-Length: 721598
< Connection: Keep-Alive
<
{ [5 bytes data]
100 704k 100 704k 0 0 7968k 0 --:--:-- --:--:-- --:--:-- 8007k
* Connection #0 to host localhost left intact
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #364771 (closed)