Research and evaluate database migration from AWS Aurora to RDS
With the latest update to always enable the database load balancer (!68042 (merged)) that will be released in %14.4, there are some PostgreSQL functions that will run for all instances which seem to not be supported by AWS Aurora (discussion thread).
We are planning to test both Amazon Aurora and Amazon RDS for PostgreSQL to validate whether our assumption about Aurora is correct and make sure that RDS does not have those issues (gitlab-org/quality/testcases#257 (comment 691633150)).
If our assumption is correct and GitLab will no longer be able to work with Aurora if we keep the load balancing update in its current form, we would like to explore what the alternatives that are available to us are.
One of the alternatives is to keep the update as is (if, for example, there is no clear path to support Aurora) and offer instructions for all instances using AWS Aurora on how to migrate to a pure RDS Postgres, so that they'll be able to upgrade to GitLab 14.4 and higher versions.
For that to be a valid option, the migration should be risk free and as seamless as possible. In this issue, we would like to figure out if that is possible, test and evaluate the process and document our findings.
Related issues: #220617 (closed), gitlab-org/quality/testcases#257
Result of the investigation
The AWS documentation provides ways to migrate data both by using the AWS Database Migration Service and setting Amazon Aurora as a source and by using PostgreSQL database migration tools such as pg_dump
There is a vague section on data type conversion during the migration when the AWS Database Migration Service is used, but that may not be an issue when migrating between the same PostgreSQL versions.
If we are going to provide guidance on this, we should clarify with the AWS team that migrating by using the AWS Database Migration Service will not convert any data types, as that would cause a lot of problems for GitLab instances.
Otherwise, we can only propose using PostgreSQL database migration tools such as pg_dump.