Do not retrigger a sync pipeline when there is one running
What does this MR do and why?
Describe in detail what your merge request does and why.
Content
This change check if a sync pipeline is running and do not trigger a new pipeline in such case. It is to avoid overloading the server, which is already slow - the reason why the previous pipeline is still running.
Ref: gitlab-com/gl-infra/delivery#20576
Testing
When the pipeline succeeds
[1] pry(main)> ReleaseTools::Security::Finalize::SyncSecurityToCanonical.new.execute
2024-10-25 15:57:00.269800 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Starting to sync gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master
2024-10-25 15:57:00.269873 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Taking ownership of the pipeline schedule -- {:project=>"dat.tang.gitlab/test-project", :pipeline_schedule=>454}
2024-10-25 15:57:00.701696 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Deactivating pipeline schedule -- {:pipeline_schedule=>"gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master", :current_active=>false, :new_active=>false}
2024-10-25 15:57:01.456439 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 15:57:02.824950 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826800"}
2024-10-25 15:57:33.150074 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826800"}
2024-10-25 15:58:03.546970 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"success", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826800"}
2024-10-25 15:58:03.547102 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Successfully synced gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master
2024-10-25 15:58:03.547256 I [security] ReleaseTools::Slack::ReleaseJobEndNotifier -- Posting slack message -- {:job_type=>"Sync from security to canonical", :status=>:success}
=> {"ok"=>true,
"channel"=>"C047XHV7PQQ",
"ts"=>"1729864683.883209",
"message"=>
{"user"=>"U038VKHEWDN",
"type"=>"message",
"ts"=>"1729864683.883209",
"bot_id"=>"B037Z751MNJ",
"app_id"=>"A0385PGTMSQ",
"text"=>"Sync from security to canonical job successfully executed - <http://job.com/>",
"team"=>"T02592416",
"bot_profile"=>
{"id"=>"B037Z751MNJ",
"app_id"=>"A0385PGTMSQ",
"name"=>"Release-Tools",
"icons"=>
{"image_36"=>"https://avatars.slack-edge.com/2022-04-07/3381895602368_6f294b6e6a7bde5dcd73_36.png",
"image_48"=>"https://avatars.slack-edge.com/2022-04-07/3381895602368_6f294b6e6a7bde5dcd73_48.png",
"image_72"=>"https://avatars.slack-edge.com/2022-04-07/3381895602368_6f294b6e6a7bde5dcd73_72.png"},
"deleted"=>false,
"updated"=>1649332946,
"team_id"=>"T02592416"},
"blocks"=>
When the pipeline fails
[1] pry(main)> ReleaseTools::Security::Finalize::SyncSecurityToCanonical.new.execute
2024-10-25 16:11:24.908380 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Starting to sync gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master
2024-10-25 16:11:24.908425 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Taking ownership of the pipeline schedule -- {:project=>"dat.tang.gitlab/test-project", :pipeline_schedule=>455}
2024-10-25 16:11:25.358938 I [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Deactivating pipeline schedule -- {:pipeline_schedule=>"gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master", :current_active=>false, :new_active=>false}
2024-10-25 16:11:25.807039 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:11:28.395080 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826866"}
2024-10-25 16:11:58.775884 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826866"}
2024-10-25 16:12:03.781857 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:12:04.929493 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826867"}
2024-10-25 16:12:35.528241 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826867"}
2024-10-25 16:12:40.532953 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:12:42.079109 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826868"}
2024-10-25 16:13:12.637029 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826868"}
2024-10-25 16:13:17.642762 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:13:19.228845 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"pending", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826869"}
2024-10-25 16:13:49.561823 I [security] ReleaseTools::Services::TriggerPipelineService -- Check the triggered pipeline status -- {:status=>"running", :pipeline=>"https://ops.gitlab.net/dat.tang.gitlab/test-project/-/pipelines/3826869"}
2024-10-25 16:13:54.564882 I [security] ReleaseTools::Services::TriggerPipelineService -- Triggering pipeline schedule -- {:project=>"dat.tang.gitlab/test-project"}
2024-10-25 16:14:16.765183 F [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Failed to sync gitlab-org/security/gitlab@master -> gitlab-org/gitlab@master -- {:error=>#<Errno::ETIMEDOUT: Operation timed out - SSL_connect>}
2024-10-25 16:14:16.765590 F [security] ReleaseTools::Security::Finalize::SyncSecurityToCanonical -- Manual action is required:
- Disable the `merge_train_to_canonical` feature flag on ops: https://ops.gitlab.net/gitlab-org/release/tools/-/feature_flags.
- Enable the `gitlab-org/gitlab@master -> gitlab-org/security/gitlab@master` pipeline schedule on the merge-train: https://ops.gitlab.net/gitlab-org/merge-train/-/pipeline_schedules.
- Execute the `sync_remotes` task on Slack: `/chatops run release sync_remotes --security`. In this case, if the sync fails, a merge request will be created and release manager intervention will be required.
2024-10-25 16:14:16.765858 I [security] ReleaseTools::Slack::ReleaseJobEndNotifier -- Posting slack message -- {:job_type=>"Sync from security to canonical", :status=>:failed}
Edited by Dat Tang