Replicating objects in object storage schedules removal and re-syncs
Summary
Using GitLab managed object storage replication with Geo, objects are being periodically deleted and re-synchronized.
Steps to reproduce
- Have a Geo setup with different object storages set up
- Enable replicating object storage
- Push different LFS objects and observe previous ones getting re-synchronized (not 100% sure of the "trigger", this was initially observed on a customer's instance)
What is the current bug behavior?
Objects get removed and re-synchronized.
What is the expected correct behavior?
Objects should not get re-synchronized if they are already present.
Relevant logs and/or screenshots
{"severity":"INFO","time":"2020-02-07T00:15:28.468Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::MigratedLocalFilesCleanUpWorker","message":"Scheduled Geo::FileRegistryRemovalWorker","job_id":"****ad304f3e828","id":11,"type":"lfs"}
{"severity":"INFO","time":"2020-02-07T00:15:28.470Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::MigratedLocalFilesCleanUpWorker","message":"Scheduled Geo::FileRegistryRemovalWorker","job_id":"****96d9904a497","id":4653,"type":"file"}
{"severity":"INFO","time":"2020-02-07T00:15:28.472Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::FileRegistryRemovalWorker","message":"Executing Geo::FileRegistryRemovalService","job_id":"****ad304f3e828","id":11,"type":"lfs","file_path":null}
{"severity":"INFO","time":"2020-02-07T00:15:28.473Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::FileRegistryRemovalService","object_type":"lfs","object_db_id":11,"message":"Executing"}
{"severity":"INFO","time":"2020-02-07T00:15:28.473Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::MigratedLocalFilesCleanUpWorker","message":"Scheduled Geo::FileRegistryRemovalWorker","job_id":"****8428f75d9db","id":386,"type":"job_artifact"}
{"severity":"INFO","time":"2020-02-07T00:15:28.473Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::FileRegistryRemovalWorker","message":"Executing Geo::FileRegistryRemovalService","job_id":"****96d9904a497","id":4653,"type":"file","file_path":null}
{"severity":"INFO","time":"2020-02-07T00:15:28.474Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::FileRegistryRemovalService","object_type":"lfs","object_db_id":11,"message":"Lease obtained"}
{"severity":"INFO","time":"2020-02-07T00:15:28.474Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::FileRegistryRemovalService","object_type":"file","object_db_id":4653,"message":"Executing"}
{"severity":"INFO","time":"2020-02-07T00:15:28.476Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::FileRegistryRemovalService","object_type":"file","object_db_id":4653,"message":"Lease obtained"}
{"severity":"INFO","time":"2020-02-07T00:15:28.476Z","correlation_id":"e678861356d****f8621d09e","class":"Geo::MigratedLocalFilesCleanUpWorker","message":"Scheduled Geo::FileRegistryRemovalWorker","job_id":"****48b0b1631fb","id":13,"type":"lfs"}
Output of checks
Results of GitLab environment info
Happens on 12.7.2-ee, reported by customer in ZD (internal-only).
Possible fixes
Possible culprit might be the MigratedLocalFilesCleanUpWorker
- which can potentially "find" all objects for removal.
/cc @vsizov