Skip to content

Added zip+zstd and tar+zstd archivers

Arran Walker requested to merge ajwalker/add-tarzstd-support into main

What does this MR do?

This MR does two things:

  • Registers zipzstd zip archiver support. This is to future proof GitLab-Runner, so that should it produce zip artifacts with the zstd compression algorithm in the future, it will be supported. Having this backwards compatibility early is good, as it eases the rollout of such a feature. zstd for zip files is now supported by many zip utilities.
  • Adds tarzstd for caches. GitLab-Runner caches are internal to Runner only, so we can be more flexible on the format used. This introduced tar and zstd support which should have lower memory requirements.

Job's can use CACHE_COMPRESSION_FORMAT with zip (default) or tarzstd. If a pipeline uses tarzstd, all runners in that pipeline will need to also have support, as the archive format won't be supported by older runners.

Why was this MR needed?

Zstd is better than the deflate based archives we support today.

  1. It doesn't cost much to implement.
  2. By registering it as supported, GitLab can later support zipzstd archives easily, and this will be backwards compatible with Runner from this point forward. More and more zip utilities are starting to support zstd, and the earlier it is in Runner, the easier the transition can be.

What's the best way to test this MR?

Some of the existing archiving tests have been modified.

Best manual QA.

What are the relevant issue numbers?

Edited by Arran Walker

Merge request reports