Enable TLS support for Patroni API
What does this MR do?
This MR:
- enables TLS on Patroni API server (see: https://patroni.readthedocs.io/en/latest/SETTINGS.html#rest-api)
- enables client TLS authentication in
gitlab-ctl patroni ...
andpatronictl
Testing
Users and passwords
First run the following script to set up the required passwords (and their hash):
alias pg-password-md5="echo 'require \"openssl\"; puts Digest::MD5.hexdigest(\"#{ARGV[1]}#{ARGV[0]}\")' | ruby -- - "
export GITLAB_SQL_USER='gitlab'
export GITLAB_SQL_PASSWORD='secret'
export GITLAB_SQL_PASSWORD_HASH="$(pg-password-md5 ${GITLAB_SQL_USER} ${GITLAB_SQL_PASSWORD})"
export PGBOUNCER_SQL_USER='pgbouncer'
export PGBOUNCER_SQL_PASSWORD='secret'
export PGBOUNCER_SQL_PASSWORD_HASH="$(pg-password-md5 ${PGBOUNCER_SQL_USER} ${PGBOUNCER_SQL_PASSWORD})"
export REPLICATOR_SQL_USER='gitlab_replicator'
export REPLICATOR_SQL_PASSWORD='secret'
export REPLICATOR_SQL_PASSWORD_HASH="$(pg-password-md5 ${REPLICATOR_SQL_USER} ${REPLICATOR_SQL_PASSWORD})"
export CONSUL_USER='gitlab-consul'
export CONSUL_PASSWORD='secret'
export CONSUL_PASSWORD_HASH="$(pg-password-md5 ${CONSUL_USER} ${CONSUL_PASSWORD})"
Select build Docker image and replica counts
Use Docker EE image from the build pipeline:
# The registry requires authentication
export TEST_IMAGE='registry.gitlab.com/gitlab-org/build/omnibus-gitlab-mirror/gitlab-ee'
# Update with the latest build tag
export TEST_TAG='14.1.1-rfbranch.345323818.3a65c338-0'
export CONSUL_REPLICA_COUNT=3
export PATRONI_REPLICA_COUNT=3
Generate TLS certificates
I use step
which is simpler than openssl
:
Note: Certificate SAN depends on the number of Consul and Patroni replicas for. Docker Compose generally allocates container IP addresses sequentially. The following script works under this assumption.
SAN="$(printf '--san 172.28.28.%s ' $(seq $((CONSUL_REPLICA_COUNT + 2)) $((CONSUL_REPLICA_COUNT + PATRONI_REPLICA_COUNT + 1))))"
mkdir -p ./tls/
echo 'secret' > tls/password.txt
step certificate create \
--profile root-ca \
--password-file tls/password.txt \
patroni-ca tls/ca.crt.pem tls/ca.key.pem
step certificate create \
--password-file tls/password.txt \
--ca ./tls/ca.crt.pem \
--ca-key ./tls/ca.key.pem \
--ca-password-file tls/password.txt \
--san patroni \
--san localhost \
--san 127.0.0.1 \
$(echo $SAN) \
patroni ./tls/patroni.crt.pem ./tls/patroni.key.pem
step certificate create \
--ca ./tls/ca.crt.pem \
--ca-key ./tls/ca.key.pem \
--ca-password-file tls/password.txt \
--insecure \
--no-password \
--san patroni \
--san localhost \
--san 127.0.0.1 \
$(echo $SAN) \
patroni ./tls/client.crt.pem ./tls/client.key.pem
# This is a terrible thing to do but required for testing:
chmod 755 ./tls
chmod 644 ./tls/*.pem
Run a test cluster
Use the following Docker Compose file and to set up Consul and Patroni services:
docker-compose -p gitlab up --scale patroni=${PATRONI_REPLICA_COUNT} --scale consul=${CONSUL_REPLICA_COUNT} -d
version: "3.9"
networks:
test:
driver: bridge
ipam:
driver: default
config:
- subnet: 172.28.28.0/24
services:
consul:
image: '${TEST_IMAGE}:${TEST_TAG}'
networks:
test:
volumes:
- ${PWD}/tls:/opt/tls:ro
environment:
GITLAB_OMNIBUS_CONFIG: |
consul['enable'] = true
consul['configuration'] = {
server: true,
client_addr: '0.0.0.0',
bootstrap_expect: ${CONSUL_REPLICA_COUNT},
retry_join: %w(consul)
}
gitlab_rails['auto_migrate'] = false
gitlab_exporter['enable'] = false
sidekiq['enable'] = false
puma['enable'] = false
registry['enable'] = false
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
prometheus_monitoring['enable'] = false
redis['enable'] = false
patroni:
depends_on:
- consul
image: '${TEST_IMAGE}:${TEST_TAG}'
networks:
test:
volumes:
- ${PWD}/tls:/opt/tls:ro
environment:
GITLAB_OMNIBUS_CONFIG: |
roles ['patroni_role']
postgresql['listen_address'] = '0.0.0.0'
postgresql['pgbouncer_user_password'] = '${PGBOUNCER_SQL_PASSWORD_HASH}'
postgresql['sql_user_password'] = '${GITLAB_SQL_PASSWORD_HASH}'
postgresql['sql_replication_password'] = '${REPLICATOR_SQL_PASSWORD}'
postgresql['trust_auth_cidr_addresses'] = %w(127.0.0.1/32 172.28.28.0/24)
postgresql['md5_auth_cidr_addresses'] = %w(127.0.0.1/32 172.28.28.0/24)
consul['services'] = %w(postgresql)
consul['configuration'] = {
retry_join: %w(consul)
}
patroni['consul']['url'] = 'http://consul:8500'
patroni['tls_verify'] = false
patroni['tls_certificate_file'] = '/opt/tls/patroni.crt.pem'
patroni['tls_key_file'] = '/opt/tls/patroni.key.pem'
patroni['tls_key_password'] = 'secret'
#patroni['tls_ca_file'] = '/opt/tls/ca.crt.pem'
#patroni['tls_client_mode'] = 'required'
#patroni['tls_client_certificate'] = '/opt/tls/client.crt.pem'
#patroni['tls_client_key'] = '/opt/tls/client.key.pem'
gitlab_rails['auto_migrate'] = false
gitlab_exporter['enable'] = false
sidekiq['enable'] = false
puma['enable'] = false
registry['enable'] = false
gitaly['enable'] = false
gitlab_workhorse['enable'] = false
nginx['enable'] = false
prometheus_monitoring['enable'] = false
redis['enable'] = false
Test scenarios
Change the following configuration and test with gitlab-ctl patroni ...
commands or curl https://patroni:8008 ...
.
patroni['tls_verify'] = false
patroni['tls_certificate_file'] = '/opt/tls/patroni.crt.pem'
patroni['tls_key_file'] = '/opt/tls/patroni.key.pem'
patroni['tls_key_password'] = 'secret'
#patroni['tls_ca_file'] = '/opt/tls/ca.crt.pem'
#patroni['tls_client_mode'] = 'required'
#patroni['tls_client_certificate'] = '/opt/tls/client.crt.pem'
#patroni['tls_client_key'] = '/opt/tls/client.key.pem'
Related issues
Closes #6261 (closed)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion
Required
-
Merge Request Title, and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com -
Pipeline is green on dev.gitlab.org if the change is touching anything besides documentation or internal cookbooks -
trigger-package
has a green pipeline running against latest commit
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updated: gitlab!67290 (merged) -
Tests added -
Integration tests added to GitLab QA -
Equivalent MR/issue for the GitLab Chart opened
Edited by Hossein Pursultani