Geo: Implement backfill of PackageFiles/Blobs

This should require very little PackageFile-specific code, since replication of new PackageFiles with events should already be implemented.

The true goal is implementation of Self-Service Framework and its documentation.

Proposal

• RegistryConsistencyWorker creates missing PackageFileRegistry records (this worker already exists, just add PackageFileRegistry to its list, for now) => #217863 (closed)
• Another worker (e.g. RegistrySyncWorker, I can't think of a reason why it would be Blob-specific) enqueues sync jobs for registries needing sync
  • This is currently handled by e.g. FileDownloadDispatchWorker, and we should copy as much as we need, but I strongly recommend not reusing it as-is.
  • This is our best opportunity to change that logic without risk.
    • Use a worker that doesn't run long, e.g. Reenqueuer
    • We now have a Replicator "interface".
    • Sync concurrency settings should apply across event-triggered syncs as well, not just backfill ones, so it should not be managed solely within this worker. Instead, concurrency limits should be handled further down, in the worker doing syncing. So, for this issue we can ignore concurrency limits. Later, we can stop enqueuing jobs from this worker if we are already at the concurrency limit.
  • Optionally, make it sync only "unsynced" in 1 MR, then also "failed" and "missing on primary" in another MR.
• We can reuse Geo::EventWorker as sync jobs, see https://gitlab.com/gitlab-org/gitlab/-/blob/v12.9.2-ee/ee/lib/gitlab/geo/log_cursor/events/event.rb#L11: ::Geo::EventWorker.perform_async(event.replicable_name, event.event_name, event.payload). E.g. ::Geo::EventWorker.perform_async("package_file", "created", { "model_record_id" => 4 })
• Anything that needs to be done in order to add a new blob must be added to https://docs.gitlab.com/ee/development/geo/framework.html#blob-replicator-strategy. After following the doc, there should be sufficient specs in place that guarantee Geo support of the new blob.