Introducing LFK CleanupWorker turbo mode

What does this MR do and why?

Introducing LFK CleanupWorker turbo mode behind a FF.

Addressing: #419119 (closed)

Changelog: fixed

Update regarding this MR

We will put this on hold and try to deliver !128759 (merged) first and see the results. We might not need a turbo mode after all, but we can still merge it in case it was needed in the future.

Why are we implementing this

The LooseForeignKeys::CleanupWorker is currently not catching up with the piled up deleted records on the ci database. We want to introduce the turbo mode which is enabled via OPS FF, so that we can increase the limits temporarily in case this happens.

We have another issue to add alerting to this status of the LFK deleted records: #419580

How to set up and validate locally

1. Stop the sidekiq workers so that they don't mess up with the testing.

gdk stop rails-background-jobs

2. Prepare Data

Run this in the Rails Console

pipeline = FactoryBot.create(:ci_pipeline, project: Project.last); nil
builds = FactoryBot.create_list(:ci_build, 50, pipeline: pipeline, upstream_pipeline_id: pipeline.id);
builds.each do |build|
  FactoryBot.create(:security_scan, build: build); nil
end; nil
pipeline.destroy; nil # This will create a LFK Deleted Record in the CI database. You can check via LooseForeignKeys::DeletedRecord.using_connection(Ci::ApplicationRecord.connection) { LooseForeignKeys::DeletedRecord.count }

3. Prepare the code

We need to monkey patch the constants, to make it easy for us to test without creating thousands of test records. Also, we will force the LooseForeignKeys::CleanupWorker to always use the ci database, because that's what we are testing only. main and ci have the exact code.

To do this, run this code in the Rails Console

module LooseForeignKeys
  # rubocop: disable CodeReuse/ActiveRecord
  class CleanerService
    DELETE_LIMIT = 1
    UPDATE_LIMIT = 1
  end
end  

module LooseForeignKeys
  class ModificationTracker
    def max_deletes; 5; end
    def max_updates; 5; end
  end
end

module LooseForeignKeys
  class TurboModificationTracker
    def max_deletes; 10; end
    def max_updates; 10; end
  end
end

module LooseForeignKeys
  class CleanupWorker
    def current_connection_name_and_base_model
      ["ci", Ci::ApplicationRecord]
    end
  end
end

4. Testing

Run this in the Rails Console

With the Turbo FF set to false

This code should delete 5 records, and should output turbo: false. This goes into the logs for debugging as well. It should print something that looks like {:delete_count => 5, turbo_mode: false}

Feature.disable(:loose_foreign_keys_turbo_mode_ci)
LooseForeignKeys::CleanupWorker.new.perform

With the Turbo FF set to true, the delete_count should reach 10 and turbo_mode should be true.

Feature.enable(:loose_foreign_keys_turbo_mode_ci)
LooseForeignKeys::CleanupWorker.new.perform

5. Cleaning up

Disable the FF

Feature.disable(:loose_foreign_keys_turbo_mode_ci)

and enable Sidekiq again

gdk start rails-background-jobs

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #419119 (closed)