Skip to content

Do not retrigger a sync pipeline when there is one running

Dat Tang requested to merge dattang/correct-exponential-retry-delay-time into master

What does this MR do and why?

Describe in detail what your merge request does and why.

Content

This change check if a sync pipeline is running and do not trigger a new pipeline in such case. It is to avoid overloading the server, which is already slow - the reason why the previous pipeline is still running.

Ref: gitlab-com/gl-infra/delivery#20576

Testing

When the pipeline succeeds

[1] pry(main)> ReleaseTools::Security::Finalize::SyncSecurityToCanonical.new.execute
2024-10-25 15:57:00.269800 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Starting to sync  gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master
2024-10-25 15:57:00.269873 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Taking ownership of the pipeline schedule -- {:project=>"dat.tang.gitlab/test-project", :pipeline_schedule=>454}
2024-10-25 15:57:00.701696 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Deactivating pipeline schedule -- {:pipeline_schedule=>"gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master", :current_active=>false, :new_active=>false}
2024-10-25 15:57:01.456439 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 15:57:02.824950 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826800"}
2024-10-25 15:57:33.150074 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826800"}
2024-10-25 15:58:03.546970 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"success", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826800"}
2024-10-25 15:58:03.547102 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Successfully synced gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master
2024-10-25 15:58:03.547256 I [security] ReleaseTools::Slack::ReleaseJobEndNotifier -- Posting slack message -- {:job_type=>"Sync from security to canonical", :status=>:success}
=> {"ok"=>true,
 "channel"=>"C047XHV7PQQ",
 "ts"=>"1729864683.883209",
 "message"=>
  {"user"=>"U038VKHEWDN",
   "type"=>"message",
   "ts"=>"1729864683.883209",
   "bot_id"=>"B037Z751MNJ",
   "app_id"=>"A0385PGTMSQ",
   "text"=>"Sync from security to canonical job successfully executed - <http://job.com/>",
   "team"=>"T02592416",
   "bot_profile"=>
    {"id"=>"B037Z751MNJ",
     "app_id"=>"A0385PGTMSQ",
     "name"=>"Release-Tools",
     "icons"=>
      {"image_36"=>"https://avatars.slack-edge.com/2022-04-07/3381895602368_6f294b6e6a7bde5dcd73_36.png",
       "image_48"=>"https://avatars.slack-edge.com/2022-04-07/3381895602368_6f294b6e6a7bde5dcd73_48.png",
       "image_72"=>"https://avatars.slack-edge.com/2022-04-07/3381895602368_6f294b6e6a7bde5dcd73_72.png"},
     "deleted"=>false,
     "updated"=>1649332946,
     "team_id"=>"T02592416"},
   "blocks"=>

When the pipeline fails

[1] pry(main)> ReleaseTools::Security::Finalize::SyncSecurityToCanonical.new.execute
2024-10-25 16:11:24.908380 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Starting to sync  gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master
2024-10-25 16:11:24.908425 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Taking ownership of the pipeline schedule -- {:project=>"dat.tang.gitlab/test-project", :pipeline_schedule=>455}
2024-10-25 16:11:25.358938 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Deactivating pipeline schedule -- {:pipeline_schedule=>"gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master", :current_active=>false, :new_active=>false}
2024-10-25 16:11:25.807039 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:11:28.395080 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826866"}
2024-10-25 16:11:58.775884 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826866"}
2024-10-25 16:12:03.781857 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:12:04.929493 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826867"}
2024-10-25 16:12:35.528241 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826867"}
2024-10-25 16:12:40.532953 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:12:42.079109 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826868"}
2024-10-25 16:13:12.637029 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826868"}
2024-10-25 16:13:17.642762 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:13:19.228845 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826869"}
2024-10-25 16:13:49.561823 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826869"}
2024-10-25 16:13:54.564882 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:14:16.765183 F [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Failed to sync gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master -- {:error=>#<Errno::ETIMEDOUT: Operation timed out - SSL_connect>}
2024-10-25 16:14:16.765590 F [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Manual action is required:
- Disable the `merge_train_to_canonical` feature flag on ops: https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags.
- Enable the `gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master` pipeline schedule on the merge-train: https://ops.gitlab.net/gitlab-org/merge-train/-/pipeline_schedules.
- Execute the `sync_remotes` task on Slack: `/chatops run release sync_remotes --security`. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.

2024-10-25 16:14:16.765858 I [security] ReleaseTools::Slack::ReleaseJobEndNotifier -- Posting slack message -- {:job_type=>"Sync from security to canonical", :status=>:failed}
Edited by Dat Tang

Merge request reports