refactor(storage): changed upload purging algorithm to reduce memory usage for very large registries
Related to #216
Implementation of "Pipelines" solution from #216 issue.
I changed getOutstandingUploads
return signature to work with a channel of results/errors (outstandingUpload
struct) which causes tiny test and PurgeUploads
changes. Errors and results channels were combined in one results channel which made it possible to simplify inner fetching logic. I preserve syncMap, but added delete
method to free used space as soon as possible. Also I delete uploadData
constructor because we don't need pre-set startedAt
field because by new logic we don't send uploadData
with zero startedAt field.
Still think this approach is worse than "Database" described in #216. Maybe if i understand the database data structure better, a can go back and try to implement "Database" approach