Geo: Make RegistryConsistencyWorker clean up LFS object registries
Using LFS as an example because this follows on !20002 (merged).
How do we ensure LFS objects are synced to a secondary?
- Create events (after we migrate LFS syncing onto to the new framework)
But how do we guarantee it?
-
RegistryConsistencyWorker
ensures registries exist when they should, eventually. Including after selective sync changes or enablingsync_object_storage
. -
FileDownloadDispatchWorker
syncs registries marked unsynced or failed - Verification resyncs registries if the data does not match (after we migrate LFS syncing onto to the new framework)
How do we ensure LFS objects are removed from a secondary?
-
LfsObjectDeletedEvent
deletes LFS objects and registries - Upon changing selective sync settings,
RepositoriesChangedEvent
is created... (but ATM we only use it to clean up repositories, we don't do anything for other data types) - Upon disabling
sync_object_storage
, we should send an event, but we don't currently
But how do we guarantee it?
- I propose we extend
RegistryConsistencyWorker
to ensure registries are removed (and remove files if possible)
Proposal
RegistryConsistencyWorker
already must traverse both tables, so it can trigger removals as well. This way we have some process for ensuring eventual consistency of registry tables with a high degree of confidence. Events are fast, but relatively less robust.
-
Also remove the Remove entry
button from the UI since this handles it automatically
Unresolved questions
- If
RegistryConsistencyWorker
comes across a registry that does not have a corresponding model record because the model record was deleted, we don't know where the file is in order to delete it. Maybe an event will later delete the file and registry-- or maybe that job was lost. I've been thinking maybe we should store file location information on registries so we don't have to derive it from the read-replica DB + the codebase. Data should not just move around, so the duplication of information seems safe. If primary data does move around, then at least we have a shot at moving/removing the old data in this case. Without storing the information, there is much more risk of orphaning data with little to no recourse. - Is it correct to remove files from object storage when
sync_object_storage
becomes disabled? It feels like we could get into a weird situation in this case.
Edited by Douglas Barbosa Alexandre