Skip to content

Run single DB jobs when CI or Geo models changed

Context

Closes #396297 (closed)

What does this MR do?

We run single-db CI jobs when CI/Geo models are changed in an MR.

How many single-db jobs would be run?

A concern was raised in #385168 (comment 1269171917),

My concern is that I am not very sure if this might make it run too many jobs for those related changes, but it's hard to tell without looking at the actual data (or just try it on production ™)

If we look at https://gitlab.com/gitlab-org/gitlab/-/pipelines/804558400, we are running more than 200 single-db jobs (100 of them are coming from 4 jobs: rspec system pg12 * and rspec unit pg12 *).

How often would we run those jobs?

It's hard to give an exact frequency, but we have 44 model files under geo folders and 83 model files under ci folders:

Click to expand
 Dir.glob("{,ee/,jh/}app/models/{ci,geo}/**/*")
=> 
["app/models/ci/application_record.rb",
 "app/models/ci/artifact_blob.rb",
 "app/models/ci/bridge.rb",
 "app/models/ci/build.rb",
 "app/models/ci/build_dependencies.rb",
 "app/models/ci/build_metadata.rb",
 "app/models/ci/build_need.rb",
 "app/models/ci/build_pending_state.rb",
 "app/models/ci/build_report_result.rb",
 "app/models/ci/build_runner_session.rb",
 "app/models/ci/build_trace.rb",
 "app/models/ci/build_trace_chunk.rb",
 "app/models/ci/build_trace_chunks",
 "app/models/ci/build_trace_chunks/database.rb",
 "app/models/ci/build_trace_chunks/fog.rb",
 "app/models/ci/build_trace_chunks/redis.rb",
 "app/models/ci/build_trace_chunks/redis_base.rb",
 "app/models/ci/build_trace_chunks/redis_trace_chunks.rb",
 "app/models/ci/build_trace_metadata.rb",
 "app/models/ci/catalog",
 "app/models/ci/catalog/listing.rb",
 "app/models/ci/catalog/resource.rb",
 "app/models/ci/commit_with_pipeline.rb",
 "app/models/ci/daily_build_group_report_result.rb",
 "app/models/ci/deleted_object.rb",
 "app/models/ci/freeze_period.rb",
 "app/models/ci/group.rb",
 "app/models/ci/group_variable.rb",
 "app/models/ci/instance_variable.rb",
 "app/models/ci/job_artifact.rb",
 "app/models/ci/job_token",
 "app/models/ci/job_token/allowlist.rb",
 "app/models/ci/job_token/project_scope_link.rb",
 "app/models/ci/job_token/scope.rb",
 "app/models/ci/job_variable.rb",
 "app/models/ci/namespace_mirror.rb",
 "app/models/ci/namespace_settings.rb",
 "app/models/ci/partition.rb",
 "app/models/ci/pending_build.rb",
 "app/models/ci/persistent_ref.rb",
 "app/models/ci/pipeline.rb",
 "app/models/ci/pipeline_artifact.rb",
 "app/models/ci/pipeline_chat_data.rb",
 "app/models/ci/pipeline_config.rb",
 "app/models/ci/pipeline_message.rb",
 "app/models/ci/pipeline_metadata.rb",
 "app/models/ci/pipeline_schedule.rb",
 "app/models/ci/pipeline_schedule_variable.rb",
 "app/models/ci/pipeline_variable.rb",
 "app/models/ci/processable.rb",
 "app/models/ci/project_mirror.rb",
 "app/models/ci/ref.rb",
 "app/models/ci/resource.rb",
 "app/models/ci/resource_group.rb",
 "app/models/ci/runner.rb",
 "app/models/ci/runner_machine.rb",
 "app/models/ci/runner_machine_build.rb",
 "app/models/ci/runner_namespace.rb",
 "app/models/ci/runner_project.rb",
 "app/models/ci/runner_version.rb",
 "app/models/ci/running_build.rb",
 "app/models/ci/secure_file.rb",
 "app/models/ci/sources",
 "app/models/ci/sources/pipeline.rb",
 "app/models/ci/stage.rb",
 "app/models/ci/trigger.rb",
 "app/models/ci/trigger_request.rb",
 "app/models/ci/unit_test.rb",
 "app/models/ci/unit_test_failure.rb",
 "app/models/ci/variable.rb",
 "ee/app/models/ci/minutes",
 "ee/app/models/ci/minutes/additional_pack.rb",
 "ee/app/models/ci/minutes/context.rb",
 "ee/app/models/ci/minutes/cost_setting.rb",
 "ee/app/models/ci/minutes/namespace_monthly_usage.rb",
 "ee/app/models/ci/minutes/notification.rb",
 "ee/app/models/ci/minutes/project_monthly_usage.rb",
 "ee/app/models/ci/minutes/quota.rb",
 "ee/app/models/ci/minutes/usage.rb",
 "ee/app/models/ci/sources",
 "ee/app/models/ci/sources/project.rb",
 "ee/app/models/ci/subscriptions",
 "ee/app/models/ci/subscriptions/project.rb",
 "ee/app/models/geo/base_registry.rb",
 "ee/app/models/geo/cache_invalidation_event.rb",
 "ee/app/models/geo/ci_secure_file_registry.rb",
 "ee/app/models/geo/ci_secure_file_state.rb",
 "ee/app/models/geo/container_repository_registry.rb",
 "ee/app/models/geo/container_repository_state.rb",
 "ee/app/models/geo/deleted_project.rb",
 "ee/app/models/geo/dependency_proxy_blob_registry.rb",
 "ee/app/models/geo/dependency_proxy_blob_state.rb",
 "ee/app/models/geo/dependency_proxy_manifest_registry.rb",
 "ee/app/models/geo/dependency_proxy_manifest_state.rb",
 "ee/app/models/geo/design_registry.rb",
 "ee/app/models/geo/event.rb",
 "ee/app/models/geo/event_log.rb",
 "ee/app/models/geo/event_log_state.rb",
 "ee/app/models/geo/group_wiki_repository_registry.rb",
 "ee/app/models/geo/hashed_storage_attachments_event.rb",
 "ee/app/models/geo/hashed_storage_migrated_event.rb",
 "ee/app/models/geo/job_artifact_registry.rb",
 "ee/app/models/geo/job_artifact_state.rb",
 "ee/app/models/geo/lfs_object_registry.rb",
 "ee/app/models/geo/lfs_object_state.rb",
 "ee/app/models/geo/merge_request_diff_registry.rb",
 "ee/app/models/geo/package_file_registry.rb",
 "ee/app/models/geo/pages_deployment_registry.rb",
 "ee/app/models/geo/pages_deployment_state.rb",
 "ee/app/models/geo/pipeline_artifact_registry.rb",
 "ee/app/models/geo/project_registry.rb",
 "ee/app/models/geo/project_wiki_repository_registry.rb",
 "ee/app/models/geo/project_wiki_repository_state.rb",
 "ee/app/models/geo/push_user.rb",
 "ee/app/models/geo/repositories_changed_event.rb",
 "ee/app/models/geo/repository_created_event.rb",
 "ee/app/models/geo/repository_deleted_event.rb",
 "ee/app/models/geo/repository_renamed_event.rb",
 "ee/app/models/geo/repository_updated_event.rb",
 "ee/app/models/geo/reset_checksum_event.rb",
 "ee/app/models/geo/secondary_usage_data.rb",
 "ee/app/models/geo/snippet_repository_registry.rb",
 "ee/app/models/geo/terraform_state_version_registry.rb",
 "ee/app/models/geo/tracking_base.rb",
 "ee/app/models/geo/upload_registry.rb",
 "ee/app/models/geo/upload_state.rb",
 "ee/app/models/geo/wiki_repository_state.rb"]

Ideas on how to reduce the number of jobs

The absolute number of jobs for parallel jobs will go down once we're down with one of our KRs.

For this MR specifically, we might want to only run single-db for certain jobs only (e.g. not for migrations, just for rspec).

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by David Dieulivol

Merge request reports

Loading