Geo: Make Job Artifacts replication use Self-Service Framework
What does this MR do and why?
Migrate Job Artifacts replication to the new Geo SSF framework. It also adds Job Artifacts verification.
Database Migrations
Up:
== 20211119111006 CreateJobArtifactStates: migrating ==========================
-- create_table(:job_artifact_states, {:id=>false})
-- quote_column_name(:verification_failure)
-> 0.0000s
-> 0.0091s
== 20211119111006 CreateJobArtifactStates: migrated (0.0092s) =================
Geo database:
== 20210111911002 PrepareJobArtifactRegistryForSsf: migrating ================
-- change_column_default(:job_artifact_registry, :retry_count, {:from=>nil, :to=>0})
-> 0.0022s
-- add_column(:job_artifact_registry, :last_synced_at, :datetime_with_timezone)
-> 0.0006s
-- add_column(:job_artifact_registry, :verified_at, :datetime_with_timezone)
-> 0.0005s
-- add_column(:job_artifact_registry, :verification_started_at, :datetime_with_timezone)
-> 0.0005s
-- add_column(:job_artifact_registry, :verification_retry_at, :datetime_with_timezone)
-> 0.0005s
-- add_column(:job_artifact_registry, :state, :integer, {:null=>false, :limit=>2, :default=>0})
-> 0.0009s
-- add_column(:job_artifact_registry, :verification_state, :integer, {:default=>0, :null=>false, :limit=>2})
-> 0.0010s
-- add_column(:job_artifact_registry, :verification_retry_count, :integer, {:default=>0, :limit=>2, :null=>false})
-> 0.0009s
-- add_column(:job_artifact_registry, :verification_checksum, :binary)
-> 0.0006s
-- add_column(:job_artifact_registry, :verification_checksum_mismatched, :binary)
-> 0.0006s
-- add_column(:job_artifact_registry, :checksum_mismatch, :boolean, {:default=>false, :null=>false})
-> 0.0009s
-- add_column(:job_artifact_registry, :verification_failure, :string, {:limit=>255})
-> 0.0006s
-- add_column(:job_artifact_registry, :last_sync_failure, :string, {:limit=>255})
-> 0.0006s
-- transaction_open?()
-> 0.0001s
-- index_exists?(:job_artifact_registry, :verification_retry_at, {:name=>:job_artifact_registry_failed_verification, :order=>"NULLS FIRST", :where=>"((state = 2) AND (verification_state = 3))", :algorithm=>:concurrently})
-> 0.0024s
-- add_index(:job_artifact_registry, :verification_retry_at, {:name=>:job_artifact_registry_failed_verification, :order=>"NULLS FIRST", :where=>"((state = 2) AND (verification_state = 3))", :algorithm=>:concurrently})
-> 0.0013s
-- transaction_open?()
-> 0.0000s
-- index_exists?(:job_artifact_registry, :verification_state, {:name=>:job_artifact_registry_needs_verification, :where=>"((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))", :algorithm=>:concurrently})
-> 0.0029s
-- add_index(:job_artifact_registry, :verification_state, {:name=>:job_artifact_registry_needs_verification, :where=>"((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))", :algorithm=>:concurrently})
-> 0.0013s
-- transaction_open?()
-> 0.0000s
-- index_exists?(:job_artifact_registry, :verified_at, {:name=>:job_artifact_registry_pending_verification, :order=>"NULLS FIRST", :where=>"((state = 2) AND (verification_state = 0))", :algorithm=>:concurrently})
-> 0.0032s
-- add_index(:job_artifact_registry, :verified_at, {:name=>:job_artifact_registry_pending_verification, :order=>"NULLS FIRST", :where=>"((state = 2) AND (verification_state = 0))", :algorithm=>:concurrently})
-> 0.0013s
== 20210111911002 PrepareJobArtifactRegistryForSsf: migrated (0.0283s) =======
== 20210111911110 FixStateColumnInJobArtifactRegistry: migrating =============
-- transaction_open?()
-> 0.0000s
-- exec_query("SELECT COUNT(*) AS count FROM \"job_artifact_registry\" WHERE \"job_artifact_registry\".\"success\" = TRUE")
-> 0.0010s
== 20210111911110 FixStateColumnInJobArtifactRegistry: migrated (0.0015s) ====
Down:
== 20211119111006 CreateJobArtifactStates: reverting ==========================
-- drop_table(:job_artifact_states)
-> 0.0034s
== 20211119111006 CreateJobArtifactStates: reverted (0.0035s) =================
Geo database:
== 20210111911002 PrepareJobArtifactRegistryForSsf: reverting ================
-- transaction_open?()
-> 0.0000s
-- index_exists?(:job_artifact_registry, :verification_retry_at, {:name=>:job_artifact_registry_failed_verification, :algorithm=>:concurrently})
-> 0.0058s
-- remove_index(:job_artifact_registry, {:name=>:job_artifact_registry_failed_verification, :algorithm=>:concurrently, :column=>:verification_retry_at})
-> 0.0052s
-- transaction_open?()
-> 0.0000s
-- index_exists?(:job_artifact_registry, :verification_state, {:name=>:job_artifact_registry_needs_verification, :algorithm=>:concurrently})
-> 0.0034s
-- remove_index(:job_artifact_registry, {:name=>:job_artifact_registry_needs_verification, :algorithm=>:concurrently, :column=>:verification_state})
-> 0.0042s
-- transaction_open?()
-> 0.0000s
-- index_exists?(:job_artifact_registry, :verified_at, {:name=>:job_artifact_registry_pending_verification, :algorithm=>:concurrently})
-> 0.0030s
-- remove_index(:job_artifact_registry, {:name=>:job_artifact_registry_pending_verification, :algorithm=>:concurrently, :column=>:verified_at})
-> 0.0038s
== 20210111911002 PrepareJobArtifactRegistryForSsf: reverted (0.0301s) =======
== 20210111911110 FixStateColumnInJobArtifactRegistry: reverting =============
== 20210111911110 FixStateColumnInJobArtifactRegistry: reverted (0.0000s) ====
Job db:gitlabcom-database-testing - https://gitlab.com/gitlab-org/gitlab/-/jobs/2007833782
Database review notes
This MR is based on template changes from .gitlab/issue_templates/Geo Replicate a new blob type.md
document. So
basically, all the changes have been reviewed many times for the rest of the similar tables but let's see if we can spot anything else.
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
The only thing needed to enable the new feature is FF enabling:
- Enable the invite modal
Feature.enable(:geo_job_artifact_replication)
- Check how Geo replicates Job Artifact in admin area. -->
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #327400 (closed)