Fix ActiveRecord::StaleObjectError where pipeline is updated twice
What does this MR do and why?
The Ci::PipelineProcessing::AtomicProcessingService is responsible for updating the status of a pipeline and stages based on the status of jobs.
Thus everywhere in the application until recently we canceled cancelable builds by running pipeline.cancel_running and relied on the code in AtomicProcessingService to update the pipeline and stage to canceled based on the status of the builds:
https://gitlab.com/gitlab-org/gitlab/-/blob/master/lib/gitlab/ci/status/composite.rb#L49
In the graphql controller only we introduced calling pipeline.cancel directly to update the pipeline which can cause a conflict between the updates happening in the Ci::PipelineProcessing::AtomicProcessingService and what's happening in the graphql controller. Since we don't lock retry and rescue the stale object error this causes the error to bubble up as a flash message sometimes, although otherwise the behavior appears normal. For some reason cascade exacerbates this (based on some staging tests). However no errors occur via the non graphql api.
Either way I think we should rely on a single way of canceling pipelines via cancel_running, which will become a service someday.
We may have to open an issue to fix the bug that is mentioned here(but I need more details): !84774 (merged). It's possible this was added to make a feature spec pass that was not using sidekiq_inline. That hypothisis is based on the comment that is was blocking an MR that was front end only but where the feature spec was added testing the pipeline was canceled without :sidekiq_inline. Without sidekiq logic running the AtomicProcessingService the feature spec does not work.
Stack trace(see app/graphql/mutations/ci/pipeline/cancel.rb:16:):
"exception.backtrace": [
"lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'",
"lib/gitlab/database/load_balancing/load_balancer.rb:191:in `retry_with_backoff'",
"lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'",
"app/models/concerns/cross_database_modification.rb:93:in `block in transaction'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'",
"lib/gitlab/database/load_balancing/load_balancer.rb:115:in `block in read_write'",
"lib/gitlab/database/load_balancing/load_balancer.rb:191:in `retry_with_backoff'",
"lib/gitlab/database/load_balancing/load_balancer.rb:111:in `read_write'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'",
"lib/gitlab/database/load_balancing/connection_proxy.rb:71:in `transaction'",
"lib/gitlab/database.rb:331:in `block in transaction'",
"lib/gitlab/database.rb:330:in `transaction'",
"app/models/concerns/cross_database_modification.rb:84:in `transaction'",
"app/graphql/mutations/ci/pipeline/cancel.rb:16:in `resolve'",
"lib/gitlab/graphql/present/field_extension.rb:18:in `resolve'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:20:in `block in trace'",
"lib/gitlab/application_context.rb:103:in `block in use'",
"lib/gitlab/application_context.rb:103:in `use'",
"lib/gitlab/application_context.rb:48:in `with_context'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:19:in `trace'",
"lib/gitlab/graphql/tracers/timer_tracer.rb:20:in `trace'",
"lib/gitlab/graphql/generic_tracing.rb:48:in `with_labkit_tracing'",
"lib/gitlab/graphql/generic_tracing.rb:38:in `platform_trace'",
"lib/gitlab/graphql/tracers/logger_tracer.rb:14:in `trace'",
"lib/gitlab/graphql/tracers/metrics_tracer.rb:13:in `trace'",
"lib/gitlab/graphql/tracers/application_context_tracer.rb:23:in `trace'",
"app/graphql/gitlab_schema.rb:51:in `multiplex'",
"app/controllers/graphql_controller.rb:167:in `execute_query'",
"app/controllers/graphql_controller.rb:57:in `execute'",
"ee/lib/gitlab/ip_address_state.rb:10:in `with'",
"ee/app/controllers/ee/application_controller.rb:45:in `set_current_ip_address'",
"app/controllers/application_controller.rb:531:in `set_current_admin'",
"lib/gitlab/session.rb:11:in `with_session'",
"app/controllers/application_controller.rb:522:in `set_session_storage'",
"lib/gitlab/i18n.rb:107:in `with_locale'",
"lib/gitlab/i18n.rb:113:in `with_user_locale'",
"app/controllers/application_controller.rb:516:in `set_locale'",
"app/controllers/application_controller.rb:510:in `set_current_context'",
"ee/lib/omni_auth/strategies/group_saml.rb:41:in `other_phase'",
"lib/gitlab/metrics/elasticsearch_rack_middleware.rb:16:in `call'",
"lib/gitlab/middleware/memory_report.rb:13:in `call'",
"lib/gitlab/middleware/speedscope.rb:13:in `call'",
"lib/gitlab/database/load_balancing/rack_middleware.rb:23:in `call'",
"lib/gitlab/middleware/rails_queue_duration.rb:33:in `call'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `block in call'",
"lib/gitlab/metrics/web_transaction.rb:46:in `run'",
"lib/gitlab/metrics/rack_middleware.rb:16:in `call'",
"lib/gitlab/jira/middleware.rb:19:in `call'",
"lib/gitlab/middleware/go.rb:20:in `call'",
"lib/gitlab/etag_caching/middleware.rb:21:in `call'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `block in call'",
"lib/gitlab/database/query_analyzer.rb:37:in `within'",
"lib/gitlab/middleware/query_analyzer.rb:11:in `call'",
"lib/gitlab/middleware/multipart.rb:173:in `call'",
"lib/gitlab/middleware/read_only/controller.rb:50:in `call'",
"lib/gitlab/middleware/read_only.rb:18:in `call'",
"lib/gitlab/middleware/same_site_cookies.rb:27:in `call'",
"lib/gitlab/middleware/handle_malformed_strings.rb:21:in `call'",
"lib/gitlab/middleware/basic_health_check.rb:25:in `call'",
"lib/gitlab/middleware/handle_ip_spoof_attack_error.rb:25:in `call'",
"lib/gitlab/middleware/request_context.rb:21:in `call'",
"lib/gitlab/middleware/webhook_recursion_detection.rb:15:in `call'",
"config/initializers/fix_local_cache_middleware.rb:11:in `call'",
"lib/gitlab/middleware/compressed_json.rb:26:in `call'",
"lib/gitlab/middleware/rack_multipart_tempfile_factory.rb:19:in `call'",
"lib/gitlab/middleware/sidekiq_web_static.rb:20:in `call'",
"lib/gitlab/metrics/requests_rack_middleware.rb:77:in `call'",
"lib/gitlab/middleware/release_env.rb:13:in `call'"
],