Simplify/optimize the assets compilation jobs and update caches on 2-hourly scheduled master pipelines [UPDATE CACHE]

What does this MR do?

Simplify and optimize the assets compilation jobs

1. Cache `node_modules` for these jobs.
1. De-duplicate the `pull-cache` and `pull-push-cache` jobs.

Update caches in specific jobs on 2-hourly scheduled master pipelines

Also update caches when MR title includes `UPDATE CACHES` (for testing
purpose mostly).

3. Don't warn about missing secrets in the test environment (so that warnings don't make static-analysis fail).

Timing improvements

1. Cache is only pulled in regular jobs and updated only on 2-hourly scheduled master pipelines.
2. setup-test-env (critical path) will finish 1 minute 6 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231038, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560478880):
   • Uploading cache: 4 seconds instead of 1 minute 10 seconds => 1 minute 6 seconds saved. 🚀
3. compile-test-assets (critical path) will finish 1 minute 59 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231035, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560478970):
   • Restoring cache: 1 minute 57 seconds instead of 2 minutes 51 seconds => 54 seconds saved. 🚀
   • Uploading artifacts: 24 seconds instead of 1 minute 29 seconds => 1 minute 5 seconds saved. 🚀
4. frontend-fixtures (critical path) will finish 1 minute 59 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231047, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560432741):
   • Downloading artifacts: 46 seconds instead of 1 minute 36 seconds => 50 seconds saved. 🚀
   • Uploading artifacts: 3 seconds instead of 1 minute 12 seconds => 1 minute 9 seconds saved. 🚀
5. static-analysis will start 9 minutes earlier and finish 1 minute 13 seconds faster (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231089, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560597629):
   • Restoring cache: 1 minute 35 seconds instead of 45 seconds => 50 seconds lost. 🐢
   • Downloading artifacts: 4 seconds instead of 1 minute 36 seconds => 1 minute 32 seconds saved. 🚀
   • Uploading cache: 4 seconds instead of 35 seconds => 31 seconds saved. 🚀
6. jest will finish 3 minutes 17 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560271177, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560268012):
   • Restoring cache: 50 seconds instead of 1 minutes 36 seconds => 46 seconds saved. 🚀
   • Downloading artifacts: 3 seconds instead of 1 minute 11 seconds => 1 minute 8 seconds saved. 🚀
   • Uploading cache: 0 seconds instead of 1 minute 23 seconds => 1 minute 23 seconds saved. 🚀
7. jest-integration will finish 1 minute 56 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231061, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560432750):
   • Restoring cache: 4 seconds instead of 50 seconds => 46 seconds saved. 🚀
   • Downloading artifacts: 3 seconds instead of 1 minute 10 seconds => 1 minute 7 seconds saved. 🚀
   • Uploading cache: 3 seconds instead of 6 seconds => 3 seconds saved. 🚀
8. karma will finish 1 minutes 50 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231055, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560432745):
   • Restoring cache: 50 seconds instead of 1 minutes 32 seconds => 42 seconds saved. 🚀
   • Downloading artifacts: 4 seconds instead of 1 minute 12 seconds => 1 minute 8 seconds saved. 🚀
9. qa-frontend-node:* will finish 5 minutes 57 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231064, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560432754):
   • Restoring cache: 51 seconds instead of 2 minutes 56 seconds => 2 minutes 5 seconds saved. 🚀
   • Uploading cache: 3 seconds instead of 3 minute 55 seconds => 3 minute 52 seconds saved. 🚀
10. downtime_check will start 8 minutes earlier and finish 26 seconds faster.
11. compile-production-assets will finish 52 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231024, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560616276):
    • Restoring cache: 1 minute 51 seconds instead of 2 minutes 43 seconds => 52 seconds saved. 🚀
12. webpack-dev-server will finish 50 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231070, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560587115):
    • Restoring cache: 51 seconds instead of 54 seconds => 3 seconds saved. 🚀
    • Uploading cache: 3 seconds instead of 50 seconds => 47 seconds saved. 🚀
13. rspec jobs will finish 50 seconds earlier (before: https://gitlab.com/gitlab-org/gitlab/-/jobs/560231182, after: https://gitlab.com/gitlab-org/gitlab/-/jobs/560432922):
    • Downloading artifacts: 46 seconds instead of 1 minute 36 seconds => 50 seconds saved. 🚀

Cost improvements

1. New jobs on 2-hourly master scheduled pipelines:
   • update-yarn-cache: 3 minutes 21 seconds
   • update-assets-compile-test-cache: 19 minutes
   • update-assets-compile-production-cache: 50 minutes?
   • update-rails-cache: 15 minutes 49 seconds
   • update-static-analysis-cache: 31 minutes 46 seconds
2. Deleted jobs from master pipelines:
   • gitlab:assets:compile pull-push-cache (all master pipelines): 25 minutes
   • compile-assets pull-push-cache (all master pipelines): 12 minutes
   • compile-assets pull-push-cache as-if-foss (push and 2-hourly master pipelines): 11 minutes

• Assuming 1448 master pipelines per month (https://app.periscopedata.com/app/gitlab/496118/Engineering-Productivity-Sandbox?widget=8625910&udv=785399), including:
  • 30 nightly pipeline, 1 per day
  • 12*30 = 360 2-hourly pipelines, 12 per day
  • 1448 - (30 + 360) = 1058 push pipelines, approximately 35 per day
• Assuming 150 frontend pipelines per day on average (https://app.periscopedata.com/app/gitlab/564156/Engineering-Productivity---Pipeline?widget=8722399&udv=895976).
• Assuming 300 code pipelines per day on average (https://app.periscopedata.com/app/gitlab/564156/Engineering-Productivity---Pipeline?widget=8722399&udv=895976).
• Assuming 1 CI minute costs $0.95 / 60 = $0.016.

Added CI minutes for 2-hourly master scheduled pipelines:

• (3 + 19 + 50 + 16 + 32) * 12 * 0.016 = $23 per day, $690 per month.

Removed CI minutes for all master pipelines (i.e. gitlab:assets:compile pull-push-cache and compile-assets pull-push-cache)

• (25 + 12) * (1 + 12 + 35) * 0.016 = $28.4 per day, $852 per month.
• [qa-frontend-node:*] 6 (minutes) * 2 (jobs) * (1 + 12 + 35) * 0.016 = $9.2 per day, $276 per month

Removed CI minutes for push and 2-hourly master pipelines (i.e. compile-assets pull-push-cache as-if-foss)

• 11 * (12 + 35) * 0.016 = $8.3 per day, $249 per month.

Removed CI minutes per frontend pipeline (i.e. with all the jobs that were improved, excluding qa-frontend-node:* since it only runs on master)

• (66 + 119 + 119 + 73 + 197 + 116 + 110 + 26 + 52 + 50 + 50 = 978 seconds = 16 minutes) * 150 * 0.016 = $38.4 per day = $768 per month (counting only week days)
• (66 + 119 + 119 + 73 + 197 + 116 + 110 + 26 + 52 + 50 + 50 = 978 seconds = 16 minutes) * 300 * 0.016 = $76.8 per day = $1536 per month (counting only week days)

Net cost saving per month

852 + 276 + 249 + 768 + 1536 - 690 = $2991 per month. This is a very rough estimate but should give a good idea of the potential expected savings.

Does this MR meet the acceptance criteria?

Conformity

• Documentation (if required)
• Code review guidelines
• Merge request performance guidelines
• Style guides
• Separation of EE specific content

Related to #216730 (closed).

Closes #218440 (closed)