Geo: Migrate design repositories replication/verification to use SSF

Replicate Design Management Repositories - Repository

This issue is for implementing Geo replication and verification of Design Management Repositories.

For more background, see Geo self-service framework.

In order to implement and test this feature, you need to first set up Geo locally.

There are three main sections below. It is a good idea to structure your merge requests this way as well:

Modify database schemas to prepare to add Geo support for Design Management Repositories
Implement Geo support of Design Management Repositories behind a feature flag
Release Geo support of Design Management Repositories

It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the Proof of Concept to replicate Pipeline Artifacts.

You can look into the following example for implementing replication/verification for a new Git repository type:

Add snippet repository verification

Modify database schemas to prepare to add Geo support for Design Management Repositories

Add the registry table to track replication and verification state

Geo secondary sites have a Geo tracking database independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.

Create the migration file in ee/db/geo/migrate:

bin/rails generate migration CreateDesignManagementRepositoryRegistry --database geo

Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on design_management_repository_id because the design_management_repositories table is in a different database. The application code must handle logic such as propagating deletions.

# frozen_string_literal: true

class CreateDesignManagementRepositoryRegistry < Gitlab::Database::Migration[2.1]
  def change
    create_table :design_management_repository_registry, id: :bigserial, force: :cascade do |t|
      t.bigint :design_management_repository_id, null: false
      t.datetime_with_timezone :created_at, null: false
      t.datetime_with_timezone :last_synced_at
      t.datetime_with_timezone :retry_at
      t.datetime_with_timezone :verified_at
      t.datetime_with_timezone :verification_started_at
      t.datetime_with_timezone :verification_retry_at
      t.integer :state, default: 0, null: false, limit: 2
      t.integer :verification_state, default: 0, null: false, limit: 2
      t.integer :retry_count, default: 0, limit: 2, null: false
      t.integer :verification_retry_count, default: 0, limit: 2, null: false
      t.boolean :checksum_mismatch, default: false, null: false
      t.boolean :force_to_redownload, default: false, null: false
      t.boolean :missing_on_primary, default: false, null: false
      t.binary :verification_checksum
      t.binary :verification_checksum_mismatched
      t.text :verification_failure, limit: 255
      t.text :last_sync_failure, limit: 255

      t.index :design_management_repository_id, name: :index_design_management_repository_registry_on_design_management_repository_id, unique: true
      t.index :retry_at
      t.index :state
      # To optimize performance of DesignManagementRepositoryRegistry.verification_failed_batch
      t.index :verification_retry_at,
        name: :design_management_repository_registry_failed_verification,
        order: "NULLS FIRST",
        where: "((state = 2) AND (verification_state = 3))"
      # To optimize performance of DesignManagementRepositoryRegistry.needs_verification_count
      t.index :verification_state,
        name: :design_management_repository_registry_needs_verification,
        where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))"
      # To optimize performance of DesignManagementRepositoryRegistry.verification_pending_batch
      t.index :verified_at,
        name: :design_management_repository_registry_pending_verification,
        order: "NULLS FIRST",
        where: "((state = 2) AND (verification_state = 0))"
    end
  end
end

If deviating from the above example, then be sure to order columns according to our guidelines.

Add the new table to the database dictionary defined in ee/db/docs/:

table_name: design_management_repository_registry
description: Description example
introduced_by_url: Merge request link
milestone: Milestone example
feature_categories:
 - Feature category example
classes:
 - Class example
gitlab_schema: gitlab_geo

Run Geo tracking database migrations:
```
bin/rake db:migrate:geo
```
Be sure to commit the relevant changes in ee/db/geo/structure.sql and the file under ee/db/geo/schema_migrations

Add verification state to the Model

The Geo primary site needs to checksum every replicable so secondaries can verify their own checksums. To do this, Geo requires the Model to have an associated table to track verification state.

Create the migration file in db/migrate:

bin/rails generate migration CreateDesignManagementRepositoryStates

Replace the contents of the migration file with:

# frozen_string_literal: true

class CreateDesignManagementRepositoryStates < Gitlab::Database::Migration[2.1]
  VERIFICATION_STATE_INDEX_NAME = "index_design_management_repository_states_on_verification_state"
  PENDING_VERIFICATION_INDEX_NAME = "index_design_management_repository_states_pending_verification"
  FAILED_VERIFICATION_INDEX_NAME = "index_design_management_repository_states_failed_verification"
  NEEDS_VERIFICATION_INDEX_NAME = "index_design_management_repository_states_needs_verification"

  enable_lock_retries!

  def up
    create_table :design_management_repository_states, id: false do |t|
      t.datetime_with_timezone :verification_started_at
      t.datetime_with_timezone :verification_retry_at
      t.datetime_with_timezone :verified_at
      t.references :design_management_repository, primary_key: true, default: nil, index: false, foreign_key: { on_delete: :cascade }
      t.integer :verification_state, default: 0, limit: 2, null: false
      t.integer :verification_retry_count, default: 0, limit: 2, null: false
      t.binary :verification_checksum, using: 'verification_checksum::bytea'
      t.text :verification_failure, limit: 255

      t.index :verification_state, name: VERIFICATION_STATE_INDEX_NAME
      t.index :verified_at,
        where: "(verification_state = 0)",
        order: { verified_at: 'ASC NULLS FIRST' },
        name: PENDING_VERIFICATION_INDEX_NAME
      t.index :verification_retry_at,
        where: "(verification_state = 3)",
        order: { verification_retry_at: 'ASC NULLS FIRST' },
        name: FAILED_VERIFICATION_INDEX_NAME
      t.index :verification_state,
        where: "(verification_state = 0 OR verification_state = 3)",
        name: NEEDS_VERIFICATION_INDEX_NAME
    end
  end

  def down
    drop_table :design_management_repository_states
  end
end

If deviating from the above example, then be sure to order columns according to our guidelines.
If design_management_repositories is a high-traffic table, follow the database documentation to use with_lock_retries

Add the new table to the database dictionary defined in db/docs/:

---
table_name: design_management_repository_states
description: Separate table for cool widget verification states
introduced_by_url: https://gitlab.com/gitlab-org/gitlab/-/merge_requests/XXXXX
milestone: 'XX.Y'
feature_categories:
 - geo_replication
classes:
 - Geo::DesignManagementRepositoryState
gitlab_schema: gitlab_main

Run database migrations:
```
bin/rake db:migrate
```
Be sure to commit the relevant changes in db/structure.sql and the file under db/schema_migrations

That's all of the required database changes.

Implement Geo support of Design Management Repositories behind a feature flag

Step 1. Implement replication and verification

Add the following lines to the design_management_repository model to accomplish some important tasks:

Include ::Geo::ReplicableModel in the DesignManagementRepository class, and specify the Replicator class with_replicator Geo::DesignManagementRepositoryReplicator.
Include the ::Geo::VerifiableModel concern.
Delegate verification related methods to the design_management_repository_state model.
For verification, override some scopes to use the design_management_repository_states table instead of the model table.
Implement the verification_state_object method to return the object that holds the verification details
Override some methods to use the design_management_repository_states table in verification-related queries.

Pay some attention to method pool_repository. Not every repository type uses repository pooling. As Geo prefers to use repository snapshotting, it can lead to data loss. Make sure to overwrite pool_repository so it returns nil for repositories that do not have pools.

At this point the DesignManagementRepository class should look like this:

# frozen_string_literal: true

class DesignManagementRepository < ApplicationRecord
  ...
  include ::Geo::ReplicableModel
  include ::Geo::VerifiableModel

  delegate(*::Geo::VerificationState::VERIFICATION_METHODS, to: :design_management_repository_state)

  with_replicator Geo::DesignManagementRepositoryReplicator

  has_one :design_management_repository_state, autosave: false, inverse_of: :design_management_repository, class_name: 'Geo::DesignManagementRepositoryState'

  after_save :save_verification_details

  # Override the `all` default if not all records can be replicated. For an
  # example of an existing Model that needs to do this, see
  # `EE::MergeRequestDiff`.
  # scope :available_replicables, -> { all }

  scope :available_verifiables, -> { joins(:design_management_repository_state) }

  scope :checksummed, -> {
    joins(:design_management_repository_state).where.not(design_management_repository_states: { verification_checksum: nil })
  }

  scope :not_checksummed, -> {
    joins(:design_management_repository_state).where(design_management_repository_states: { verification_checksum: nil })
  }

  scope :with_verification_state, ->(state) {
    joins(:design_management_repository_state)
      .where(design_management_repository_states: { verification_state: verification_state_value(state) })
  }

  def verification_state_object
    design_management_repository_state
  end
  ...

  class_methods do
    extend ::Gitlab::Utils::Override
    ...

    # @param primary_key_in [Range, DesignManagementRepository] arg to pass to primary_key_in scope
    # @return [ActiveRecord::Relation<DesignManagementRepository>] everything that should be synced
    #         to this node, restricted by primary key
    def replicables_for_current_secondary(primary_key_in)
      # This issue template does not help you write this method.
      #
      # This method is called only on Geo secondary sites. It is called when
      # we want to know which records to replicate. This is not easy to automate
      # because for example:
      #
      # * The "selective sync" feature allows admins to choose which namespaces
      #   to replicate, per secondary site. Most Models are scoped to a
      #   namespace, but the nature of the relationship to a namespace varies
      #   between Models.
      # * The "selective sync" feature allows admins to choose which shards to
      #   replicate, per secondary site. Repositories are associated with
      #   shards. Most blob types are not, but Project Uploads are.
      # * Remote stored replicables are not replicated, by default. But the
      #   setting `sync_object_storage` enables replication of remote stored
      #   replicables.
      #
      # Search the codebase for examples, and consult a Geo expert if needed.
    end

    override :verification_state_table_class
    def verification_state_table_class
      DesignManagementRepositoryState
    end
  end

  # Geo checks this method in FrameworkRepositorySyncService to avoid
  # snapshotting repositories using object pools
  def pool_repository
    nil
  end

  def design_management_repository_state
    super || build_design_management_repository_state
  end

  ...
end

Implement DesignManagementRepository.replicables_for_current_secondary above.
Ensure DesignManagementRepository.replicables_for_current_secondary is well-tested. Search the codebase for replicables_for_current_secondary to find examples of parameterized table specs. You may need to add more FactoryBot traits.

Add the following shared examples to ee/spec/models/ee/design_management_repository_spec.rb:

  include_examples 'a replicable model with a separate table for verification state' do
    let(:verifiable_model_record) { build(:design_management_repository) } # add extra params if needed to make sure the record is in `Geo::ReplicableModel.verifiables` scope
    let(:unverifiable_model_record) { build(:design_management_repository) } # add extra params if needed to make sure the record is NOT included in `Geo::ReplicableModel.verifiables` scope
  end

Create ee/app/replicators/geo/design_management_repository_replicator.rb. Implement the #repository method which should return a <Repository> instance, and implement the class method .model to return the DesignManagementRepository class:

# frozen_string_literal: true

module Geo
  class DesignManagementRepositoryReplicator < Gitlab::Geo::Replicator
    include ::Geo::RepositoryReplicatorStrategy
    extend ::Gitlab::Utils::Override

    def self.model
      ::DesignManagementRepository
    end

    def self.git_access_class
      ::Gitlab::GitAccessDesignManagementRepository
    end

    def self.no_repo_message
      git_access_class.error_message(:no_repo)
    end

    override :verification_feature_flag_enabled?
    def self.verification_feature_flag_enabled?
      # We are adding verification at the same time as replication, so we
      # don't need to toggle verification separately from replication. When
      # the replication feature flag is off, then verification is also off
      # (see `VerifiableReplicator.verification_enabled?`)
      true
    end

    override :housekeeping_enabled?
    def self.housekeeping_enabled?
      # Remove this method if the new Git repository type supports git
      # repository housekeeping and the ::DesignManagementRepository#git_garbage_collect_worker_klass
      # is implemented. If the data type requires any action to be performed
      # before running the housekeeping override the `before_housekeeping` method
      # (see `RepositoryReplicatorStrategy#before_housekeeping`)
      false
    end

    def repository
      model_record.repository
    end
  end
end

Make sure Geo push events are created. Usually it needs some change in the app/workers/post_receive.rb file. Example:

def replicate_design_management_repository_changes(design_management_repository)
  if ::Gitlab::Geo.primary?
    design_management_repository.replicator.handle_after_update if design_management_repository
  end
end

See app/workers/post_receive.rb for more examples.

Make sure the repository removal is also handled. You may need to add something like the following in the destroy service of the repository:
```
design_management_repository.replicator.handle_after_destroy if design_management_repository.repository
```
Make sure a Geo secondary site can request and download Design Management Repositories on the Geo primary site. You may need to make some changes to Gitlab::GitAccessDesignManagementRepository. For example, see this change for Group-level Wikis.
Make sure a Geo secondary site can replicate Design Management Repositories where repository does not exist on the Geo primary site. The only way to know about this is to parse the error text. You may need to make some changes to Gitlab::DesignManagementRepositoryReplicator.no_repo_message to return the proper error message. For example, see this change for Group-level Wikis.
Generate the feature flag definition files by running the feature flag commands and following the command prompts:
```
bin/feature-flag --ee geo_design_management_repository_replication --type development --group 'group::geo'
```

Add this replicator class to the method replicator_classes in ee/lib/gitlab/geo.rb:

REPLICATOR_CLASSES = [
  ::Geo::PackageFileReplicator,
  ::Geo::DesignManagementRepositoryReplicator
]

Create ee/spec/replicators/geo/design_management_repository_replicator_spec.rb and perform the necessary setup to define the model_record variable for the shared examples:

# frozen_string_literal: true

require 'spec_helper'

RSpec.describe Geo::DesignManagementRepositoryReplicator, feature_category: :geo_replication do
  let(:model_record) { build(:design_management_repository) }

  include_examples 'a repository replicator'
  include_examples 'a verifiable replicator'
end

Create ee/app/models/geo/design_management_repository_registry.rb:

# frozen_string_literal: true

module Geo
  class DesignManagementRepositoryRegistry < Geo::BaseRegistry
    include ::Geo::ReplicableRegistry
    include ::Geo::VerifiableRegistry

    MODEL_CLASS = ::DesignManagementRepository
    MODEL_FOREIGN_KEY = :design_management_repository_id

    belongs_to :design_management_repository, class_name: 'DesignManagementRepository'
  end
end

Update REGISTRY_CLASSES in ee/app/workers/geo/secondary/registry_consistency_worker.rb.
Add a custom factory name if needed in def model_class_factory_name in ee/spec/support/helpers/ee/geo_helpers.rb.
Update it 'creates missing registries for each registry class' in ee/spec/workers/geo/secondary/registry_consistency_worker_spec.rb.
Add design_management_repository_registry to ActiveSupport::Inflector.inflections in config/initializers_before_autoloader/000_inflections.rb.

Create ee/spec/factories/geo/design_management_repository_registry.rb:

# frozen_string_literal: true

FactoryBot.define do
  factory :geo_design_management_repository_registry, class: 'Geo::DesignManagementRepositoryRegistry' do
    design_management_repository # This association should have data, like a file or repository
    state { Geo::DesignManagementRepositoryRegistry.state_value(:pending) }

    trait :synced do
      state { Geo::DesignManagementRepositoryRegistry.state_value(:synced) }
      last_synced_at { 5.days.ago }
    end

    trait :failed do
      state { Geo::DesignManagementRepositoryRegistry.state_value(:failed) }
      last_synced_at { 1.day.ago }
      retry_count { 2 }
      last_sync_failure { 'Random error' }
    end

    trait :started do
      state { Geo::DesignManagementRepositoryRegistry.state_value(:started) }
      last_synced_at { 1.day.ago }
      retry_count { 0 }
    end

    trait :verification_succeeded do
      verification_checksum { 'e079a831cab27bcda7d81cd9b48296d0c3dd92ef' }
      verification_state { Geo::DesignManagementRepositoryRegistry.verification_state_value(:verification_succeeded) }
      verified_at { 5.days.ago }
    end
  end
end

Create ee/spec/models/geo/design_management_repository_registry_spec.rb:

# frozen_string_literal: true

require 'spec_helper'

RSpec.describe Geo::DesignManagementRepositoryRegistry, :geo, type: :model, feature_category: :geo_replication do
  let_it_be(:registry) { create(:geo_design_management_repository_registry) }

  specify 'factory is valid' do
    expect(registry).to be_valid
  end

  include_examples 'a Geo framework registry'
  include_examples 'a Geo verifiable registry'
end

Add the following to ee/spec/factories/design_management_repositories.rb:

# frozen_string_literal: true

FactoryBot.modify do
  factory :design_management_repository do
    trait :verification_succeeded do
        with_file
        verification_checksum { 'abc' }
        verification_state { DesignManagementRepository.verification_state_value(:verification_succeeded) }
    end

    trait :verification_failed do
        with_file
        verification_failure { 'Could not calculate the checksum' }
        verification_state { DesignManagementRepository.verification_state_value(:verification_failed) }
    end
  end
end

If there is not an existing factory for the object in spec/factories/design_management_repositories.rb, wrap the traits in FactoryBot.create instead of FactoryBot.modify.

Make sure the factory also allows setting a project attribute. If the model does not have a direct relation to a project, you can use a transient attribute. Check out spec/factories/merge_request_diffs.rb for an example.

Following the example of Merge Request Diffs add a Geo::DesignManagementRepositoryState model in ee/app/models/geo/design_management_repository_state.rb:

# frozen_string_literal: true

module Geo
  class DesignManagementRepositoryState < ApplicationRecord
    include ::Geo::VerificationStateDefinition

    self.primary_key = :design_management_repository_id

    belongs_to :design_management_repository, inverse_of: :design_management_repository_state

    validates :verification_failure, length: { maximum: 255 }
    validates :verification_state, :design_management_repository, presence: true
  end
end

Add a factory for design_management_repository_state, in ee/spec/factories/geo/design_management_repository_states.rb:

# frozen_string_literal: true

FactoryBot.define do
  factory :geo_design_management_repository_state, class: 'Geo::DesignManagementRepositoryState' do
    design_management_repository

    trait :checksummed do
      verification_checksum { 'abc' }
    end

    trait :checksum_failure do
      verification_failure { 'Could not calculate the checksum' }
    end
  end
end

Add [:geo_design_management_repository_state, any] to skipped in spec/models/factories_spec.rb

Step 2. Implement metrics gathering

Metrics are gathered by Geo::MetricsUpdateWorker, persisted in GeoNodeStatus for display in the UI, and sent to Prometheus:

Add the following fields to Geo Node Status example responses in doc/api/geo_nodes.md:
- design_management_repositories_count
- design_management_repositories_checksum_total_count
- design_management_repositories_checksummed_count
- design_management_repositories_checksum_failed_count
- design_management_repositories_synced_count
- design_management_repositories_failed_count
- design_management_repositories_registry_count
- design_management_repositories_verification_total_count
- design_management_repositories_verified_count
- design_management_repositories_verification_failed_count
- design_management_repositories_synced_in_percentage
- design_management_repositories_verified_in_percentage
Add the same fields to GET /geo_nodes/status example response in ee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json.

Add the following fields to the Sidekiq metrics table in doc/administration/monitoring/prometheus/gitlab_metrics.md:

| `geo_design_management_repositories` | Gauge | XX.Y | Number of Design Management Repositories on primary | `url` |
| `geo_design_management_repositories_checksum_total` | Gauge | XX.Y | Number of Design Management Repositories to checksum on primary | `url` |
| `geo_design_management_repositories_checksummed` | Gauge | XX.Y | Number of Design Management Repositories that successfully calculated the checksum on primary | `url` |
| `geo_design_management_repositories_checksum_failed` | Gauge | XX.Y | Number of Design Management Repositories that failed to calculate the checksum on primary | `url` |
| `geo_design_management_repositories_synced` | Gauge | XX.Y | Number of syncable Design Management Repositories synced on secondary | `url` |
| `geo_design_management_repositories_failed` | Gauge | XX.Y | Number of syncable Design Management Repositories failed to sync on secondary | `url` |
| `geo_design_management_repositories_registry` | Gauge | XX.Y | Number of Design Management Repositories in the registry | `url` |
| `geo_design_management_repositories_verification_total` | Gauge | XX.Y | Number of Design Management Repositories to attempt to verify on secondary | `url` |
| `geo_design_management_repositories_verified` | Gauge | XX.Y | Number of Design Management Repositories successfully verified on secondary | `url` |
| `geo_design_management_repositories_verification_failed` | Gauge | XX.Y | Number of Design Management Repositories that failed verification on secondary | `url` |

Design Management Repository replication and verification metrics should now be available in the API, the Admin > Geo > Sites view, and Prometheus.

Step 3. Implement the GraphQL API

The GraphQL API is used by Admin > Geo > Replication Details views, and is directly queryable by administrators.

Add a new field to GeoNodeType in ee/app/graphql/types/geo/geo_node_type.rb:

field :design_management_repository_registries, ::Types::Geo::DesignManagementRepositoryRegistryType.connection_type,
      null: true,
      resolver: ::Resolvers::Geo::DesignManagementRepositoryRegistriesResolver,
      description: 'Find Design Management Repository registries on this Geo node. '\
                   'Ignored if `geo_design_management_repository_replication` feature flag is disabled.',
      alpha: { milestone: '15.5' } # Update the milestone

Add the new design_management_repository_registries field name to the expected_fields array in ee/spec/graphql/types/geo/geo_node_type_spec.rb.

Create ee/app/graphql/resolvers/geo/design_management_repository_registries_resolver.rb:

# frozen_string_literal: true

module Resolvers
  module Geo
    class DesignManagementRepositoryRegistriesResolver < BaseResolver
      type ::Types::Geo::GeoNodeType.connection_type, null: true

      include RegistriesResolver
    end
  end
end

Create ee/spec/graphql/resolvers/geo/design_management_repository_registries_resolver_spec.rb:

# frozen_string_literal: true

require 'spec_helper'

RSpec.describe Resolvers::Geo::DesignManagementRepositoryRegistriesResolver, feature_category: :geo_replication do
  it_behaves_like 'a Geo registries resolver', :geo_design_management_repository_registry
end

Create ee/app/finders/geo/design_management_repository_registry_finder.rb:

# frozen_string_literal: true

module Geo
  class DesignManagementRepositoryRegistryFinder
    include FrameworkRegistryFinder
  end
end

Create ee/spec/finders/geo/design_management_repository_registry_finder_spec.rb:

# frozen_string_literal: true

require 'spec_helper'

RSpec.describe Geo::DesignManagementRepositoryRegistryFinder, feature_category: :geo_replication do
  it_behaves_like 'a framework registry finder', :geo_design_management_repository_registry
end

Create ee/app/graphql/types/geo/design_management_repository_registry_type.rb:

# frozen_string_literal: true

module Types
  module Geo
    # rubocop:disable Graphql/AuthorizeTypes because it is included
    class DesignManagementRepositoryRegistryType < BaseObject
      graphql_name 'DesignManagementRepositoryRegistry'

      include ::Types::Geo::RegistryType

      description 'Represents the Geo replication and verification state of a design_management_repository'

      field :design_management_repository_id, GraphQL::Types::ID, null: false, description: 'ID of the Design Management Repository.'
    end
    # rubocop:enable Graphql/AuthorizeTypes
  end
end

Create ee/spec/graphql/types/geo/design_management_repository_registry_type_spec.rb:

# frozen_string_literal: true

require 'spec_helper'

RSpec.describe GitlabSchema.types['DesignManagementRepositoryRegistry'], feature_category: :geo_replication do
  it_behaves_like 'a Geo registry type'

  it 'has the expected fields (other than those included in RegistryType)' do
    expected_fields = %i[design_management_repository_id]

    expect(described_class).to have_graphql_fields(*expected_fields).at_least
  end
end

Add integration tests for providing DesignManagementRepository registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in ee/spec/requests/api/graphql/geo/registries_spec.rb:

it_behaves_like 'gets registries for', {
  field_name: 'designManagementRepositoryRegistries',
  registry_class_name: 'DesignManagementRepositoryRegistry',
  registry_factory: :geo_design_management_repository_registry,
  registry_foreign_key_field_name: 'designManagementRepositoryId'
}

Update the GraphQL reference documentation:

bundle exec rake gitlab:graphql:compile_docs

Individual Design Management Repository replication and verification data should now be available via the GraphQL API.

Step 4. Handle batch destroy

If batch destroy logic is implemented for a replicable, then that logic must be "replicated" by Geo secondaries. The easiest way to do this is use Geo::BatchEventCreateWorker to bulk insert a delete event for each replicable.

For example, if FastDestroyAll is used, then you may be able to use begin_fast_destroy and finalize_fast_destroy hooks, like we did for uploads.

Or if a special service is used to batch delete records and their associated data, then you probably need to hook into that service, like we did for job artifacts.

As illustrated by the above two examples, batch destroy logic cannot be handled automatically by Geo secondaries without restricting the way other teams perform batch destroys. It is up to you to produce Geo::BatchEventCreateWorker attributes before the records are deleted, and then enqueue Geo::BatchEventCreateWorker after the records are deleted.

Ensure that any batch destroy of this replicable is replicated to secondary sites
Regardless of implementation details, please verify in specs that when the parent object is removed, the new Geo::Event records are created:

  describe '#destroy' do
    subject { create(:design_management_repository) }

    context 'when running in a Geo primary node' do
      let_it_be(:primary) { create(:geo_node, :primary) }
      let_it_be(:secondary) { create(:geo_node) }

      it 'logs an event to the Geo event log when bulk removal is used', :sidekiq_inline do
        stub_current_geo_node(primary)

        expect { subject.project.destroy! }.to change(Geo::Event.where(replicable_name: :design_management_repository, event_name: :deleted), :count).by(1)

        payload = Geo::Event.where(replicable_name: :design_management_repository, event_name: :deleted).last.payload

        expect(payload['model_record_id']).to eq(subject.id)
        expect(payload['blob_path']).to eq(subject.relative_path)
        expect(payload['uploader_class']).to eq('DesignManagementRepositoryUploader')
      end
    end
  end

Code Review

When requesting review from database reviewers:

Include a comment mentioning that the change is based on a documented template.
replicables_for_current_secondary and available_replicables may differ per Model. If their queries are new, then add query plans to the MR description. An easy place to gather SQL queries is your GDK's log/test.log when running tests of these methods.

Release Geo support of Design Management Repositories

In the rollout issue you created when creating the feature flag, modify the Roll Out Steps:
- Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment.
- Add a step to Test replication and verification of Design Management Repositories on a non-GDK-deployment. For example, using GitLab Environment Toolkit.
- Add a step to Ping the Geo PM and EM to coordinate testing. For example, you might add steps to generate Design Management Repositories, and then a Geo engineer may take it from there.
In ee/config/feature_flags/development/geo_design_management_repository_replication.yml, set default_enabled: true

In ee/app/graphql/types/geo/geo_node_type.rb, remove the alpha option for the released type:

field :design_management_repository_registries, ::Types::Geo::DesignManagementRepositoryRegistryType.connection_type,
      null: true,
      resolver: ::Resolvers::Geo::DesignManagementRepositoryRegistriesResolver,
      description: 'Find Design Management Repository registries on this Geo node. '\
                   'Ignored if `geo_design_management_repository_replication` feature flag is disabled.',
      alpha: { milestone: '15.5' } # Update the milestone

Run bundle exec rake gitlab:graphql:compile_docs after the step above to regenerate the GraphQL docs.
Add a row for Design Management Repositories to the Data types table in Geo data types support
Add a row for Design Management Repositories to the Limitations on replication/verification table in Geo data types support. If the row already exists, then update it to show that Replication and Verification is released in the current version.

Edited May 17, 2023 by Aakriti Gupta