Add Patroni to Omnibus

What does this MR do?

This MR incorporates Patroni in GitLab Omnibus for database clustering. It does it in a very specific way which has its benefits and limitations.

In this MR, the design decisions were made to achieve the following objectives:

1. Patroni must co-exit with RepMgr.
2. Patroni must be an opt-in solution for database clustering. It implies that the user may switch between RepMgr and Patroni.
3. Database cluster configuration in gitlab.rb must be kept as simple as possible.
4. Introduction of new configuration attributes in gitlab.rb must be minimal, ideally none.

An overview of this MR

Roughly speaking, this MR has five pieces:

1. Software definitions for Patroni and Psycopg2. The software definitions use the latest bundled PostgreSQL version, i.e. postgresql_new. This is because they need, libpq, the PostgreSQL client library. This MR relies on backward-compatibility of libpq to work with older PostgreSQL versions.

2. Adjustments to Consul service registration to work with both repmgr and Patroni. In order to configure PgBouncer, Consul must work with Patroni to track changes of the cluster, in particular to answer "who is the leader?". Using a helper and predefined service checks consul::service_postgresql fills the gap and monitors PostgreSQL cluster status regardless of HA solution (Patroni or repmgr).

3. Changing PostgreSQL cookbook to work for Patroni as well. Patroni actively takes over the configuration of PostgreSQL and manages its process(es). When Patroni is enabled, some resources must be skipped, e.g. running initdb or createdb, or creating PostgreSQL user and enabling postgresql service (these are the tasks that are delegated to Patroni). Also, PostgreSQL configuration files, *.conf, must be relocated to be picked up later after Patroni bootstraps the node. All of these is to ensure that Patroni and PostgreSQL won't collide. Enabling Patroni effectively removes postgresql service from the list of GitLab services. Users must interact with Patroni to start or stop with PostgreSQL. That being said, the way that the bundled PostgreSQL is configured is left almost intact. Users can still work with postgresql[...] attributes in gitlab.rb. Only a few attributes are overridden for Patroni (namely patroni['postgresql'][...]).

4. Patroni cookbook which contains anything that is needed for configuring and running Patroni. To keep PostgreSQL configuration as is and to compensate for some of the steps that are skipped in postgresql::enable this MR leverages Patroni's custom_bootstrap and post_bootstrap features to copy PostgreSQL configuration and execute operations that require a running database instance such as creating roles and other database objects.

5. patroni sub-command for gitlab-ctl. This MR only offers the very basics that are needed for the existing implementation to work. Further commands will be added in #5115 (closed).

How to test it?

https://gitlab.com/pursultani/gitlab-patroni-test contains Terraform provisioning and configuration scripts for setting up a GitLab HA cluster with Patroni on GCP. You can use the build artifact of this MR's pipeline to run the test (GitLab token is required).

This MR was tested for both creating a new cluster and converting an existing repmgr cluster.

Documentation

The MR for the documentation can be found here: gitlab!33363 (merged)

Caveats

1. There is a known issue with conversion of standalone PostgreSQL instances. This issue can be fixed and will be addressed in #5265 (closed).

Improvements

1. Add more test for Patroni cookbook and gitlab-ctl patroni sub-command.

Closes #5112 (closed)