Cache Webpack assets in frontend assets compilation jobs
What does this MR do?
- It caches
public/assets/webpack
onmaster
: this doesn't seem to have a huge impact on assets compilation but I think it still has a small impact.🤔 - It compares the MD5 hash of assets files with the latest
master
MD5 hash of assets files and skip Webpack compilation if they're the same: this has a great impact for thegitlab:assets:compile pull-cache
job when assets haven't changed compared to latestmaster
Job | First run (no cache, no hash) | Second run (cache and hash present) |
---|---|---|
compile-assets pull-cache |
19 minutes 48 seconds | 8 minutes 7 seconds (58% faster) |
compile-assets pull-cache as-if-foss |
19 minutes 4 seconds | 8 minutes 29 seconds (58% faster) |
gitlab:assets:compile pull-cache |
39 minutes 22 seconds | 8 minutes 47 seconds (79% faster) |
Note that now that we're using "Pipelines for Merged Results", the merge commit will have more chance of including assets differences compared to the latest master
, meaning that the MD5 hash of assets files will probably often be different...
Possible future iteration
Another thing we could do is to:
- Use the MD5 hash for the assets image tag
- Try to pull the assets image
- If the image exists, no need to compile the Webpack assets nor build the image
- Otherwise, compile and build the image.
Results of adding `public/assets/webpack` to the cache
First jobs run with push-pull cache policy
-
compile-assets pull-cache
https://gitlab.com/gitlab-org/gitlab/-/jobs/451163310:Failed to extract cache
-
webpack:compile
finished in 171.169412905 seconds public/assets/webpack/: found 651 matching files
- Job finished in 20 minutes 40 seconds (note that this includes pushing the cache, which normally isn't part of this
pull-cache
job).
-
gitlab:assets:compile pull-cache
https://gitlab.com/gitlab-org/gitlab/-/jobs/451163305:Failed to extract cache
-
webpack:compile
finished in 1036.696164929 seconds public/assets/webpack/: found 1944 matching files
- Job finished in 37 minutes 2 seconds (note that this includes pushing the cache, which normally isn't part of this
pull-cache
job).
Second jobs run with push-pull cache policy
-
compile-assets pull-cache
https://gitlab.com/gitlab-org/gitlab/-/jobs/451230184:Successfully extracted cache
-
webpack:compile
finished in 81.611743018 seconds public/assets/webpack/: found 651 matching files
- Job finished in 10 minutes 12 seconds (10 minutes faster than with an empty cache).
-
gitlab:assets:compile pull-cache
https://gitlab.com/gitlab-org/gitlab/-/jobs/451230207:Successfully extracted cache
-
webpack:compile
finished in 919.892393249 seconds public/assets/webpack/: found 1944 matching files
- Job finished in 24 minutes 10 seconds (13 minutes faster that with an empty cache).
Third jobs run with pull cache policy
-
compile-assets pull-cache
https://gitlab.com/gitlab-org/gitlab/-/jobs/451267314:Successfully extracted cache
-
webpack:compile
finished in 78.105376317 seconds - Job finished in 7 minutes 56 seconds.
-
gitlab:assets:compile pull-cache
https://gitlab.com/gitlab-org/gitlab/-/jobs/451267313:Successfully extracted cache
-
webpack:compile
finished in 959.518880108 seconds - Job finished in 23 minutes 17 seconds.
Does this MR meet the acceptance criteria?
Conformity
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Separation of EE specific content
Relates to #34319 (closed).
Edited by Rémy Coutable