CI: fast initializing `fdroid build` job
Reduced the fdroid build
by two minutes!
I discovered the Getting source from Git repository step to be too time-consuming (took ~2m20s) I did some experiments and realized that the reason is mainly:
- There are too many dirs&files under metadata/, and the IO limit of the GitLab runner is about 1000 IOPS (?). which caused the git checkout to be very slow.
$find metadata/ | wc -l # it's 120692 !
- The gitlab CI uses fetch --depth=5000 to pull the last 5000 commits by default, which also slows things down.
I've optimized it as follows:
- Disable CI/CD built-in git clone/fetch (
Getting source from Git repository
) - Put the .git/ and metadata/ directories on ramdisk.
- Cache the .git/ directory
- Use git fetch --filter=tree:0 to fetch the entire commit history while only actually downloading the individual commit we want.
- Enable fastzip to speed up cache de/compression
It's worth noting that since GitLab's CI/CD cache
block runs before the script
block. In order to unpack the cache after .git/ ramdisk is loaded, I had to tarball it as gitdir.tar
.
Performance:
- Original: https://gitlab.com/yzqzss/f-droid-data/-/jobs/5840310248
- Applied this MR (
gitdir.tar
cache not hit): https://gitlab.com/yzqzss/f-droid-data/-/jobs/5840327304 - Applied this MR (
gitdir.tar
cache hit, saves about 5s): https://gitlab.com/yzqzss/f-droid-data/-/jobs/5840365592
Edited by yzqzss