Cache assets in gitlab:assets:compile
What does this MR do?
This adds a cache
directive to the gitlab:assets:compile
and compile-assets
jobs.
This improvement is based on omnibus-gitlab#2424 (comment 43074206).
Following are four different caching experiments. The second and fourth tries are mostly the same, and I could track the cache ballooning to https://github.com/rails/sprockets/issues/563.
Fifth try by caching tmp/cache/assets/sprockets
and monkey-patching Sprockets with https://github.com/rails/sprockets/compare/3.x...jmreid:no-mtime-for-digest-key
compile-assets
-
1st
compile-assets
run (11 minutes 16 seconds):-
rake:assets:precompile
finished in 72.26437519 seconds -
tmp/cache/assets/sprockets/
is currently 132M
-
-
2nd
compile-assets
run (5 minutes 20 seconds => 2.11x faster):-
rake:assets:precompile
finished in 6.456429399 seconds (11.2x faster) -
tmp/cache/assets/sprockets/
is currently 132M (same size)
-
-
3rd
compile-assets
run (5 minutes 35 seconds):-
rake:assets:precompile
finished in 6.780493447 seconds -
tmp/cache/assets/sprockets/
is currently 132M (same size)
-
Conclusion: With the Sprockets monkey-patch suggested in https://github.com/rails/sprockets/issues/563, the Sprockets assets precompilation speed is increased, with no ballooning effect since the mtime
isn't used in the cache key.
gitlab:assets:compile
-
1st
gitlab:assets:compile
run (23 minutes 55 seconds):-
rake:assets:precompile
finished in 178.490804189 seconds -
tmp/cache/assets/sprockets/
is currently 131M
-
-
2nd
gitlab:assets:compile
run (15 minutes 9 seconds => 1.57x faster):-
rake:assets:precompile
finished in 8.583158472 seconds (20.8x faster) -
tmp/cache/assets/sprockets/
is currently 131M (same size)
-
-
3rd
gitlab:assets:compile
run (14 minutes 0 seconds):-
rake:assets:precompile
finished in 6.766899414 seconds -
tmp/cache/assets/sprockets/
is currently 131M (same size)
-
Conclusion: With the Sprockets monkey-patch suggested in https://github.com/rails/sprockets/issues/563, the Sprockets assets precompilation speed is increased, with no ballooning effect since the mtime
isn't used in the cache key.
Fourth try by caching tmp/cache/assets/sprockets
-
1st
compile-assets
run:-
rake:assets:precompile
finished in 73.791743339 seconds -
tmp/cache/assets/sprockets/
is currently 132MB
-
-
2nd
compile-assets
run-
rake:assets:precompile
finished in 5.64133392 seconds (13.17x faster) -
tmp/cache/assets/sprockets/
is currently 152MB (+ 20MB for the exact same SHA/assets)
-
-
3rd
compile-assets
run-
rake:assets:precompile
finished in 5.78727085 seconds (no improvement) -
tmp/cache/assets/sprockets/
is currently 172MB (+ 20MB for the exact same SHA/assets)
-
Conclusion: The cache is improving the Sprockets precompilation speed, but it's also ballooning here, even if the assets didn't change!
gitlab:assets:compile
-
1st
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 190.122873861 seconds -
tmp/cache/assets/sprockets/
is currently 131MB
-
-
2nd
gitlab:assets:compile
run-
rake:assets:precompile
finished in 6.873365871 seconds (27.67x faster) -
tmp/cache/assets/sprockets/
is currently 151MB` (+ 20MB for the exact same SHA/assets)
-
Conclusion: The cache is improving the Sprockets precompilation speed, but it's also ballooning here, even if the assets didn't change!
Third try by caching public/assets
minus public/assets/webpack
(Sprockets compiled assets)
-
1st
compile-assets
run:-
rake:assets:precompile
finished in 74.44845465 seconds -
sprockets-assets/
is currently 58MB
-
-
2nd
compile-assets
run:-
rake:assets:precompile
finished in 69.041837927 seconds (10% faster) -
sprockets-assets/
is currently 58MB (same size)
-
-
3rd
compile-assets
run:-
rake:assets:precompile
finished in 69.830209646 seconds (no improvement) -
sprockets-assets/
is currently 58MB (same size)
-
Conclusion: Cache doesn't balloon but rake:assets:precompile
isn't faster...
gitlab:assets:compile
-
1st
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 189.936003268 seconds -
sprockets-assets/
is currently 52MB
-
-
2nd
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 190.181387912 seconds (no improvement) -
sprockets-assets/
is currently 52MB (same size)
-
-
3rd
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 183.757361696 seconds (4% faster) -
sprockets-assets/
is currently 52MB (same size)
-
Conclusion: Cache doesn't balloon but rake:assets:precompile
isn't faster...
Second try by caching tmp/cache/assets
(Sprockets cache folder)
-
1st
compile-assets
run:-
rake:assets:precompile
finished in 74.55412116 seconds -
tmp/cache/assets/
is currently 131MB
-
-
2nd
compile-assets
run:-
rake:assets:precompile
finished in 5.811031225 seconds (14.8x faster) -
tmp/cache/assets/
is currently 152MB (+ 21MB for the exact same SHA/assets)
-
-
3rd
compile-assets
run:-
rake:assets:precompile
finished in 6.313982111 seconds (11.7x faster) -
tmp/cache/assets/
is currently 172MB (+ 20MB for the exact same SHA/assets)
-
-
4th
compile-assets
run:-
rake:assets:precompile
finished in 6.51208368 seconds (11.3x faster) -
tmp/cache/assets/
is currently 192MB (+ 20MB for the exact same SHA/assets)
-
Conclusion: The cache is ballooning here, even if the assets didn't change!
gitlab:assets:compile
-
1st
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 161.426688669 seconds -
tmp/cache/assets/
is currently 131MB
-
-
2nd
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 9.140452779 seconds (17.6x faster) -
tmp/cache/assets/
is currently 151MB (+ 20MB for the exact same SHA/assets)
-
-
3rd
gitlab:assets:compile
run:-
rake:assets:precompile
finished in 9.759735945 seconds (16.5x faster) -
tmp/cache/assets/
is currently 171MB (+ 20MB for the exact same SHA/assets)
-
Conclusion: The cache is ballooning here, even if the assets didn't change!
First try below by caching tmp/cache
, and public/assets
-
First run with empty cache (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149256635). Total time: 11 minutes:
-
yarn:check
finished in 21 seconds -
gettext:po_to_json
finished in 8 seconds -
rake:assets:precompile
finished in 68 seconds -
webpack:compile
finished in 46 seconds
-
-
Second run with warm cache (same SHA) (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149263689). Total time: 9 minutes:
-
yarn:check
finished in 26 seconds -
gettext:po_to_json
finished in 7 seconds -
rake:assets:precompile
finished in 3 seconds -
webpack:compile
finished in 33 seconds
-
gitlab:assets:compile
-
First run with empty cache (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/148904529). Total time: 22 minutes:
-
webpack:compile
finished in 455 seconds
-
-
Second run with warm cache (same SHA) (https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149085002). Total time: 12 minutes.
-
webpack:compile
finished in 131 seconds (3.45x faster!)🚀
-
-
Third run with
new CompressionPlugin({ cache: true })
(https://gitlab.com/gitlab-org/gitlab-ce/-/jobs/149091616). Total time: 15 minutes:-
webpack:compile
finished in 135 seconds => this doesn't seem to be significant.
-
What are the relevant issue numbers?
Related to https://gitlab.com/gitlab-org/gitlab-ce/issues/56538.
Closes https://gitlab.com/gitlab-org/gitlab-ce/issues/56691, and maybe https://gitlab.com/gitlab-org/gitlab-ce/issues/35305?
Does this MR meet the acceptance criteria?
- [-] Changelog entry added, if necessary
- [-] Documentation created/updated via this MR
- [-] Documentation reviewed by technical writer or follow-up review issue created
- [-] Tests added for this feature/bug
- [-] Tested in all supported browsers
-
Conforms to the code review guidelines -
Conforms to the merge request performance guidelines -
Conforms to the style guides -
Conforms to the database guides - [-] Link to e2e tests MR added if this MR has Requires e2e tests label. See the Test Planning Process.
-
Security reports checked/validated by reviewer