Skip to content

Switch production drivers to standard Walk

Hayley Swimelar requested to merge sequentialize-production-drivers into 3-optimize-the-s3-storage-driver-s-walk-function

Rationale

As we previously discovered with S3 in !58 (merged), WalkParallel can strain resources (in this case API requests) when ran in an unbounded fashion. With this, it seems more appropriate to give special attention to each of these drivers before merging them with truly parallel walks into a release.

Changes

The swift, azure, gcs, and oss drivers call their Walk method in WalkParallel, rather than storagedriver.WalkFallbackParallel. Calling their own Walk method, rather than storagedriver.WalkFallback ensures that any gains made to their respective Walk functions before we've verified their concurrent behavior are passed onto WalkParallel

Caveats

The filesystem and inmemory drivers still use storagedriver.WalkFallbackParallel in their WalkParallel. The inmemory is explicitly for testing and filesystem does not depend on networked resources (baring something like nfs) and is less likely to be used in large production environments.

Edited by Stan Hu

Merge request reports