Run single DB jobs when CI or Geo models changed
Context
Closes #396297 (closed)
What does this MR do?
We run single-db
CI jobs when CI/Geo models are changed in an MR.
How many single-db jobs would be run?
A concern was raised in #385168 (comment 1269171917),
My concern is that I am not very sure if this might make it run too many jobs for those related changes, but it's hard to tell without looking at the actual data (or just try it on production ™)
If we look at https://gitlab.com/gitlab-org/gitlab/-/pipelines/804558400, we are running more than 200 single-db jobs (100 of them are coming from 4 jobs: rspec system pg12 *
and rspec unit pg12 *
).
How often would we run those jobs?
It's hard to give an exact frequency, but we have 44 model files under geo
folders and 83 model files under ci
folders:
Click to expand
Dir.glob("{,ee/,jh/}app/models/{ci,geo}/**/*")
=>
["app/models/ci/application_record.rb",
"app/models/ci/artifact_blob.rb",
"app/models/ci/bridge.rb",
"app/models/ci/build.rb",
"app/models/ci/build_dependencies.rb",
"app/models/ci/build_metadata.rb",
"app/models/ci/build_need.rb",
"app/models/ci/build_pending_state.rb",
"app/models/ci/build_report_result.rb",
"app/models/ci/build_runner_session.rb",
"app/models/ci/build_trace.rb",
"app/models/ci/build_trace_chunk.rb",
"app/models/ci/build_trace_chunks",
"app/models/ci/build_trace_chunks/database.rb",
"app/models/ci/build_trace_chunks/fog.rb",
"app/models/ci/build_trace_chunks/redis.rb",
"app/models/ci/build_trace_chunks/redis_base.rb",
"app/models/ci/build_trace_chunks/redis_trace_chunks.rb",
"app/models/ci/build_trace_metadata.rb",
"app/models/ci/catalog",
"app/models/ci/catalog/listing.rb",
"app/models/ci/catalog/resource.rb",
"app/models/ci/commit_with_pipeline.rb",
"app/models/ci/daily_build_group_report_result.rb",
"app/models/ci/deleted_object.rb",
"app/models/ci/freeze_period.rb",
"app/models/ci/group.rb",
"app/models/ci/group_variable.rb",
"app/models/ci/instance_variable.rb",
"app/models/ci/job_artifact.rb",
"app/models/ci/job_token",
"app/models/ci/job_token/allowlist.rb",
"app/models/ci/job_token/project_scope_link.rb",
"app/models/ci/job_token/scope.rb",
"app/models/ci/job_variable.rb",
"app/models/ci/namespace_mirror.rb",
"app/models/ci/namespace_settings.rb",
"app/models/ci/partition.rb",
"app/models/ci/pending_build.rb",
"app/models/ci/persistent_ref.rb",
"app/models/ci/pipeline.rb",
"app/models/ci/pipeline_artifact.rb",
"app/models/ci/pipeline_chat_data.rb",
"app/models/ci/pipeline_config.rb",
"app/models/ci/pipeline_message.rb",
"app/models/ci/pipeline_metadata.rb",
"app/models/ci/pipeline_schedule.rb",
"app/models/ci/pipeline_schedule_variable.rb",
"app/models/ci/pipeline_variable.rb",
"app/models/ci/processable.rb",
"app/models/ci/project_mirror.rb",
"app/models/ci/ref.rb",
"app/models/ci/resource.rb",
"app/models/ci/resource_group.rb",
"app/models/ci/runner.rb",
"app/models/ci/runner_machine.rb",
"app/models/ci/runner_machine_build.rb",
"app/models/ci/runner_namespace.rb",
"app/models/ci/runner_project.rb",
"app/models/ci/runner_version.rb",
"app/models/ci/running_build.rb",
"app/models/ci/secure_file.rb",
"app/models/ci/sources",
"app/models/ci/sources/pipeline.rb",
"app/models/ci/stage.rb",
"app/models/ci/trigger.rb",
"app/models/ci/trigger_request.rb",
"app/models/ci/unit_test.rb",
"app/models/ci/unit_test_failure.rb",
"app/models/ci/variable.rb",
"ee/app/models/ci/minutes",
"ee/app/models/ci/minutes/additional_pack.rb",
"ee/app/models/ci/minutes/context.rb",
"ee/app/models/ci/minutes/cost_setting.rb",
"ee/app/models/ci/minutes/namespace_monthly_usage.rb",
"ee/app/models/ci/minutes/notification.rb",
"ee/app/models/ci/minutes/project_monthly_usage.rb",
"ee/app/models/ci/minutes/quota.rb",
"ee/app/models/ci/minutes/usage.rb",
"ee/app/models/ci/sources",
"ee/app/models/ci/sources/project.rb",
"ee/app/models/ci/subscriptions",
"ee/app/models/ci/subscriptions/project.rb",
"ee/app/models/geo/base_registry.rb",
"ee/app/models/geo/cache_invalidation_event.rb",
"ee/app/models/geo/ci_secure_file_registry.rb",
"ee/app/models/geo/ci_secure_file_state.rb",
"ee/app/models/geo/container_repository_registry.rb",
"ee/app/models/geo/container_repository_state.rb",
"ee/app/models/geo/deleted_project.rb",
"ee/app/models/geo/dependency_proxy_blob_registry.rb",
"ee/app/models/geo/dependency_proxy_blob_state.rb",
"ee/app/models/geo/dependency_proxy_manifest_registry.rb",
"ee/app/models/geo/dependency_proxy_manifest_state.rb",
"ee/app/models/geo/design_registry.rb",
"ee/app/models/geo/event.rb",
"ee/app/models/geo/event_log.rb",
"ee/app/models/geo/event_log_state.rb",
"ee/app/models/geo/group_wiki_repository_registry.rb",
"ee/app/models/geo/hashed_storage_attachments_event.rb",
"ee/app/models/geo/hashed_storage_migrated_event.rb",
"ee/app/models/geo/job_artifact_registry.rb",
"ee/app/models/geo/job_artifact_state.rb",
"ee/app/models/geo/lfs_object_registry.rb",
"ee/app/models/geo/lfs_object_state.rb",
"ee/app/models/geo/merge_request_diff_registry.rb",
"ee/app/models/geo/package_file_registry.rb",
"ee/app/models/geo/pages_deployment_registry.rb",
"ee/app/models/geo/pages_deployment_state.rb",
"ee/app/models/geo/pipeline_artifact_registry.rb",
"ee/app/models/geo/project_registry.rb",
"ee/app/models/geo/project_wiki_repository_registry.rb",
"ee/app/models/geo/project_wiki_repository_state.rb",
"ee/app/models/geo/push_user.rb",
"ee/app/models/geo/repositories_changed_event.rb",
"ee/app/models/geo/repository_created_event.rb",
"ee/app/models/geo/repository_deleted_event.rb",
"ee/app/models/geo/repository_renamed_event.rb",
"ee/app/models/geo/repository_updated_event.rb",
"ee/app/models/geo/reset_checksum_event.rb",
"ee/app/models/geo/secondary_usage_data.rb",
"ee/app/models/geo/snippet_repository_registry.rb",
"ee/app/models/geo/terraform_state_version_registry.rb",
"ee/app/models/geo/tracking_base.rb",
"ee/app/models/geo/upload_registry.rb",
"ee/app/models/geo/upload_state.rb",
"ee/app/models/geo/wiki_repository_state.rb"]
Ideas on how to reduce the number of jobs
The absolute number of jobs for parallel jobs will go down once we're down with one of our KRs.
For this MR specifically, we might want to only run single-db
for certain jobs only (e.g. not for migrations
, just for rspec).
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.