Skip to content

fix: limit bucket listing to 5 files at most

Philippe Vienne requested to merge PhilippeVienne/CNG:improve_s3_read into master

What does this MR do?

On backup utility run, we face an issue of large slow start on s3 synchronization.

After investigation, we determined it comes from the s3cmd ls used to determine if the bucket exists. If you have a large bucket, the time needed to do all listing request is too long and may failed if upload occurs between.

To avoid that, we suggest to only list at most 5 files from the bucket.

Related issues

No related issues opened

Checklist

See Definition of done.

For anything in this list which will not be completed, please provide a reason in the MR discussion

Required

  • Merge Request Title, and Description are up to date, accurate, and descriptive
  • MR targeting the appropriate branch
  • MR has a green pipeline on GitLab.com

Expected (please provide an explanation if not completing)

  • Test plan indicating conditions for success has been posted and passes
  • Documentation created/updated
  • Integration tests added to GitLab QA
  • The impact any change in container size has should be evaluated

Merge request reports