Update the internal rails registry API to use primary
🔥 Problem
The Enqueuer and Guard workers both will always use the primary database due to having data_consistency: always
. This means that when a pre_import
is started it will be updated directly to the primary as expected.
The API that handles notifications coming from the registry is not guaranteed to use the primary. It first queries the current state of the given container repository, which will likely use a replica because it has not yet reached a write operation. If the pre-import and import happen very quickly, there is a chance that the import complete
notification from the registry will use a replica that has not yet been updated. It will think the container repository has a default
state instead of the true importing
state and will throw a 400 error.
We saw this happen on production:
{"message":"400 Bad request - Wrong migration state (default)"}
In the code, there is no way for a container repository to transition from any other state back to default
, so this means that the 2nd request was reading from an outdated source.
🚒 Solution
We should force the API endpoint to read from the primary using ::Gitlab::Database::LoadBalancing::Session.current.use_primary
.