Resolve "Optimize the garbage collector sweep stage for S3"
Context
Please see gitlab#31071 (closed) and #10 (closed) for context.
Results
The changes proposed here have shown encouraging results. Optimizations were performed in two main parts:
Each of these MRs has a detailed analysis of the current behaviour and benchmarks of the proposed improvements. Please look at them for more information.
Tasks
-
Avoid redundant digest cast and parse operations (!29 (merged)) -
Add blob deletion benchmarks (!41 (merged) and !42 (merged)) -
Implement bulk deletion for blobs (!31 (merged)) -
Implement concurrency for bulk deletion (!32 (merged)) -
Add manifest deletion benchmarks (!33 (merged)) -
Add error handling tests for the new S3 driver DeleteFiles method (!38 (merged)) -
Implement bulk deletion for manifests (!39 (merged)) -
Remove empty folders left behind with the local filesystem storage driver (!47 (merged)) -
Avoid stat calls for each tag being deleted (!52 (merged)) -
Improve logging (!50 (merged)) -
Detailed performance and cost analysis
Edited by João Pereira