Replicate AlertManagement::MetricImage
Replicate Alert Metric Images
This issue is for implementing Geo replication and verification of Alert Metric Images.
For more background, see Geo self-service framework.
In order to implement and test this feature, you need to first set up Geo locally.
There are three main sections below. It is a good idea to structure your merge requests this way as well:
- Modify database schemas to prepare to add Geo support for Alert Metric Images
- Implement Geo support of Alert Metric Images behind a feature flag
- Release Geo support of Alert Metric Images
It is also a good idea to first open a proof-of-concept merge request. It can be helpful for working out kinks and getting initial support and feedback from the Geo team. As an example, see the Proof of Concept to replicate Pipeline Artifacts.
You can look into the following examples of MRs for implementing replication/verification for a new blob type:
- Add db changes and add verification for MR diffs using SSF
- Verify Terraform state versions
- Verify LFS objects
Modify database schemas to prepare to add Geo support for Alert Metric Images
You might do this section in its own merge request, but it is not required.
Add the registry table to track replication and verification state
Geo secondary sites have a Geo tracking database independent of the main database. It is used to track the replication and verification state of all replicables. Every Model has a corresponding "registry" table in the Geo tracking database.
-
Create the migration file in ee/db/geo/migrate
:bin/rails generate migration CreateAlertManagement::MetricImageRegistry --database geo
-
Replace the contents of the migration file with the following. Note that we cannot add a foreign key constraint on alert_management_alert_metric_image_id
because thealert_management_alert_metric_images
table is in a different database. The application code must handle logic such as propagating deletions.# frozen_string_literal: true class CreateAlertManagement::MetricImageRegistry < Gitlab::Database::Migration[1.0] disable_ddl_transaction! def up ApplicationRecord.transaction do create_table :alert_management_alert_metric_image_registry, id: :bigserial, force: :cascade do |t| t.bigint :alert_management_alert_metric_image_id, null: false t.datetime_with_timezone :created_at, null: false t.datetime_with_timezone :last_synced_at t.datetime_with_timezone :retry_at t.datetime_with_timezone :verified_at t.datetime_with_timezone :verification_started_at t.datetime_with_timezone :verification_retry_at t.integer :state, default: 0, null: false, limit: 2 t.integer :verification_state, default: 0, null: false, limit: 2 t.integer :retry_count, default: 0, limit: 2, null: false t.integer :verification_retry_count, default: 0, limit: 2, null: false t.boolean :checksum_mismatch, default: false, null: false t.binary :verification_checksum t.binary :verification_checksum_mismatched t.text :verification_failure, limit: 255 t.text :last_sync_failure, limit: 255 t.index :alert_management_alert_metric_image_id, name: :index_alert_management_alert_metric_image_registry_on_alert_management_alert_metric_image_id, unique: true t.index :retry_at t.index :state # To optimize performance of AlertManagement::MetricImageRegistry.verification_failed_batch t.index :verification_retry_at, name: :alert_management_alert_metric_image_registry_failed_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 3))" # To optimize performance of AlertManagement::MetricImageRegistry.needs_verification_count t.index :verification_state, name: :alert_management_alert_metric_image_registry_needs_verification, where: "((state = 2) AND (verification_state = ANY (ARRAY[0, 3])))" # To optimize performance of AlertManagement::MetricImageRegistry.verification_pending_batch t.index :verified_at, name: :alert_management_alert_metric_image_registry_pending_verification, order: "NULLS FIRST", where: "((state = 2) AND (verification_state = 0))" end end end def down drop_table :alert_management_alert_metric_image_registry end end
-
If deviating from the above example, then be sure to order columns according to our guidelines. -
Add the new table to the GitLab Schema defined in ee/lib/ee/gitlab/database/gitlab_schemas.yml
.alert_management_alert_metric_image_registry: :gitlab_geo
-
Run Geo tracking database migrations: bin/rake db:migrate:geo
-
Be sure to commit the relevant changes in ee/db/geo/structure.sql
Add verification state fields on the Geo primary site
The Geo primary site needs to checksum every replicable so secondaries can verify their own checksums. To do this, Geo requires fields on the Model. Add verification state fields to a separate table. Consult a database expert if needed.
Add verification state fields to a new table
-
Create the migration file in db/migrate
:bin/rails generate migration CreateAlertManagement::MetricImageStates
-
Replace the contents of the migration file with: # frozen_string_literal: true class CreateAlertManagement::MetricImageStates < Gitlab::Database::Migration[2.0] VERIFICATION_STATE_INDEX_NAME = "index_alert_management_alert_metric_image_states_on_verification_state" PENDING_VERIFICATION_INDEX_NAME = "index_alert_management_alert_metric_image_states_pending_verification" FAILED_VERIFICATION_INDEX_NAME = "index_alert_management_alert_metric_image_states_failed_verification" NEEDS_VERIFICATION_INDEX_NAME = "index_alert_management_alert_metric_image_states_needs_verification" enable_lock_retries! def up create_table :alert_management_alert_metric_image_states, id: false do |t| t.datetime_with_timezone :verification_started_at t.datetime_with_timezone :verification_retry_at t.datetime_with_timezone :verified_at t.references :alert_management_alert_metric_image, primary_key: true, default: nil, index: false, foreign_key: { on_delete: :cascade } t.integer :verification_state, default: 0, limit: 2, null: false t.integer :verification_retry_count, limit: 2 t.binary :verification_checksum, using: 'verification_checksum::bytea' t.text :verification_failure, limit: 255 t.index :verification_state, name: VERIFICATION_STATE_INDEX_NAME t.index :verified_at, where: "(verification_state = 0)", order: { verified_at: 'ASC NULLS FIRST' }, name: PENDING_VERIFICATION_INDEX_NAME t.index :verification_retry_at, where: "(verification_state = 3)", order: { verification_retry_at: 'ASC NULLS FIRST' }, name: FAILED_VERIFICATION_INDEX_NAME t.index :verification_state, where: "(verification_state = 0 OR verification_state = 3)", name: NEEDS_VERIFICATION_INDEX_NAME end end def down drop_table :alert_management_alert_metric_image_states end end
-
If deviating from the above example, then be sure to order columns according to our guidelines. -
Add the new table to the GitLab Schema defined in lib/gitlab/database/gitlab_schemas.yml
with the databases they need to be added to.alert_management_alert_metric_image_states: :gitlab_main
-
Run database migrations: bin/rake db:migrate
-
If alert_management_alert_metric_images
is a high-traffic table, follow the database documentation to usewith_lock_retries
-
Be sure to commit the relevant changes in db/structure.sql
That's all of the required database changes.
Implement Geo support of Alert Metric Images behind a feature flag
Step 1. Implement replication and verification
-
Add the following lines to the alert_management_alert_metric_image
model to accomplish some important tasks:- Include
::Geo::ReplicableModel
in theAlertManagement::MetricImage
class, and specify the Replicator classwith_replicator Geo::AlertManagement::MetricImageReplicator
. - Include the
::Geo::VerifiableModel
concern. - Delegate verification related methods to the
alert_management_alert_metric_image_state
model. - For verification, override some scopes to use the
alert_management_alert_metric_image_states
table instead of the model table. - Implement the
verification_state_object
method to return the object that holds the verification details - Override some methods to use the
alert_management_alert_metric_image_states
table in verification-related queries.
At this point the
AlertManagement::MetricImage
class should look like this:# frozen_string_literal: true class AlertManagement::MetricImage < ApplicationRecord ... include ::Geo::ReplicableModel include ::Geo::VerifiableModel delegate(*::Geo::VerificationState::VERIFICATION_METHODS, to: :alert_management_alert_metric_image_state) with_replicator Geo::AlertManagement::MetricImageReplicator mount_uploader :file, AlertManagement::MetricImageUploader has_one :alert_management_alert_metric_image_state, autosave: false, inverse_of: :alert_management_alert_metric_image, class_name: 'Geo::AlertManagement::MetricImageState' after_save :save_verification_details scope :with_verification_state, ->(state) { joins(:alert_management_alert_metric_image_state).where(alert_management_alert_metric_image_states: { verification_state: verification_state_value(state) }) } scope :checksummed, -> { joins(:alert_management_alert_metric_image_state).where.not(alert_management_alert_metric_image_states: { verification_checksum: nil } ) } scope :not_checksummed, -> { joins(:alert_management_alert_metric_image_state).where(alert_management_alert_metric_image_states: { verification_checksum: nil } ) } scope :available_verifiables, -> { joins(:alert_management_alert_metric_image_state) } # Override the `all` default if not all records can be replicated. For an # example of an existing Model that needs to do this, see # `EE::MergeRequestDiff`. # scope :available_replicables, -> { all } def verification_state_object alert_management_alert_metric_image_state end ... class_methods do extend ::Gitlab::Utils::Override ... # @param primary_key_in [Range, AlertManagement::MetricImage] arg to pass to primary_key_in scope # @return [ActiveRecord::Relation<AlertManagement::MetricImage>] everything that should be synced to this node, restricted by primary key def self.replicables_for_current_secondary(primary_key_in) # This issue template does not help you write this method. # # This method is called only on Geo secondary sites. It is called when # we want to know which records to replicate. This is not easy to automate # because for example: # # * The "selective sync" feature allows admins to choose which namespaces # to replicate, per secondary site. Most Models are scoped to a # namespace, but the nature of the relationship to a namespace varies # between Models. # * The "selective sync" feature allows admins to choose which shards to # replicate, per secondary site. Repositories are associated with # shards. Most blob types are not, but Project Uploads are. # * Remote stored replicables are not replicated, by default. But the # setting `sync_object_storage` enables replication of remote stored # replicables. # # Search the codebase for examples, and consult a Geo expert if needed. end override :verification_state_table_class def verification_state_table_class AlertManagement::MetricImageState end end ... def alert_management_alert_metric_image_state super || build_alert_management_alert_metric_image_state end ... end
- Include
-
Implement AlertManagement::MetricImage.replicables_for_current_secondary
above. -
Ensure AlertManagement::MetricImage.replicables_for_current_secondary
is well-tested. Search the codebase forreplicables_for_current_secondary
to find examples of parameterized table specs. You may need to add moreFactoryBot
traits. -
Add the following shared examples to ee/spec/models/ee/alert_management_alert_metric_image_spec.rb
:include_examples 'a replicable model with a separate table for verification state' do let(:verifiable_model_record) { build(:alert_management_alert_metric_image) } # add extra params if needed to make sure the record is included in `available_verifiables` let(:unverifiable_model_record) { build(:alert_management_alert_metric_image) } # add extra params if needed to make sure the record is NOT included in `available_verifiables` end
-
Create ee/app/replicators/geo/alert_management_alert_metric_image_replicator.rb
. Implement the#carrierwave_uploader
method which should return aCarrierWave::Uploader
, and implement the class method.model
to return theAlertManagement::MetricImage
class:# frozen_string_literal: true module Geo class AlertManagement::MetricImageReplicator < Gitlab::Geo::Replicator include ::Geo::BlobReplicatorStrategy extend ::Gitlab::Utils::Override def self.model ::AlertManagement::MetricImage end def carrierwave_uploader model_record.file end override :verification_feature_flag_enabled? def self.verification_feature_flag_enabled? # We are adding verification at the same time as replication, so we # don't need to toggle verification separately from replication. When # the replication feature flag is off, then verification is also off # (see `VerifiableReplicator.verification_enabled?`) true end end end
-
Generate the feature flag definition fileы by running the feature flag commands and following the command prompts: bin/feature-flag --ee geo_alert_management_alert_metric_image_replication --type development --group 'group::geo' bin/feature-flag --ee geo_alert_management_alert_metric_image_verification --type development --group 'group::geo'
-
Add this replicator class to the method replicator_classes
inee/lib/gitlab/geo.rb
:REPLICATOR_CLASSES = [ ::Geo::PackageFileReplicator, ::Geo::AlertManagement::MetricImageReplicator ] end
-
Create ee/spec/replicators/geo/alert_management_alert_metric_image_replicator_spec.rb
and perform the necessary setup to define themodel_record
variable for the shared examples:# frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::AlertManagement::MetricImageReplicator do let(:model_record) { build(:alert_management_alert_metric_image) } include_examples 'a blob replicator' include_examples 'a verifiable replicator' end
-
Create ee/app/models/geo/alert_management_alert_metric_image_registry.rb
:# frozen_string_literal: true class Geo::AlertManagement::MetricImageRegistry < Geo::BaseRegistry include ::Geo::ReplicableRegistry include ::Geo::VerifiableRegistry MODEL_CLASS = ::AlertManagement::MetricImage MODEL_FOREIGN_KEY = :alert_management_alert_metric_image_id belongs_to :alert_management_alert_metric_image, class_name: 'AlertManagement::MetricImage' end
-
Update REGISTRY_CLASSES
inee/app/workers/geo/secondary/registry_consistency_worker.rb
. -
Add a custom factory name if needed in def model_class_factory_name
inee/spec/support/helpers/ee/geo_helpers.rb
. -
Update it 'creates missing registries for each registry class'
inee/spec/workers/geo/secondary/registry_consistency_worker_spec.rb
. -
Add alert_management_alert_metric_image_registry
toActiveSupport::Inflector.inflections
inconfig/initializers_before_autoloader/000_inflections.rb
. -
Create ee/spec/factories/geo/alert_management_alert_metric_image_registry.rb
:# frozen_string_literal: true FactoryBot.define do factory :geo_alert_management_alert_metric_image_registry, class: 'Geo::AlertManagement::MetricImageRegistry' do alert_management_alert_metric_image # This association should have data, like a file or repository state { Geo::AlertManagement::MetricImageRegistry.state_value(:pending) } trait :synced do state { Geo::AlertManagement::MetricImageRegistry.state_value(:synced) } last_synced_at { 5.days.ago } end trait :failed do state { Geo::AlertManagement::MetricImageRegistry.state_value(:failed) } last_synced_at { 1.day.ago } retry_count { 2 } last_sync_failure { 'Random error' } end trait :started do state { Geo::AlertManagement::MetricImageRegistry.state_value(:started) } last_synced_at { 1.day.ago } retry_count { 0 } end trait :verification_succeeded do verification_checksum { 'e079a831cab27bcda7d81cd9b48296d0c3dd92ef' } verification_state { Geo::AlertManagement::MetricImageRegistry.verification_state_value(:verification_succeeded) } verified_at { 5.days.ago } end end end
-
Create ee/spec/models/geo/alert_management_alert_metric_image_registry_spec.rb
:# frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::AlertManagement::MetricImageRegistry, :geo, type: :model do let_it_be(:registry) { create(:geo_alert_management_alert_metric_image_registry) } specify 'factory is valid' do expect(registry).to be_valid end include_examples 'a Geo framework registry' include_examples 'a Geo verifiable registry' end
-
Add the following to spec/factories/alert_management_alert_metric_images.rb
:trait(:verification_succeeded) do with_file verification_checksum { 'abc' } verification_state { AlertManagement::MetricImage.verification_state_value(:verification_succeeded) } end trait(:verification_failed) do with_file verification_failure { 'Could not calculate the checksum' } verification_state { AlertManagement::MetricImage.verification_state_value(:verification_failed) } end
-
Make sure the factory also allows setting a project
attribute. If the model does not have a direct relation to a project, you can use atransient
attribute. Check outspec/factories/merge_request_diffs.rb
for an example. -
Following the example of Merge Request Diffs add a Geo::AlertManagement::MetricImageState
model inee/app/models/ee/geo/alert_management_alert_metric_image_state.rb
:module Geo class AlertManagement::MetricImageState < ApplicationRecord include EachBatch include ::Geo::VerificationStateDefinition self.primary_key = :alert_management_alert_metric_image_id belongs_to :alert_management_alert_metric_image, inverse_of: :alert_management_alert_metric_image_state validates :verification_failure, length: { maximum: 255 } validates :verification_state, :alert_management_alert_metric_image, presence: true end end
-
Add a factory
foralert_management_alert_metric_image_state
, inee/spec/factories/geo/alert_management_alert_metric_image_states.rb
:# frozen_string_literal: true FactoryBot.define do factory :geo_alert_management_alert_metric_image_state, class: 'Geo::AlertManagement::MetricImageState' do alert_management_alert_metric_image trait(:checksummed) do verification_checksum { 'abc' } end trait(:checksum_failure) do verification_failure { 'Could not calculate the checksum' } end end end
Step 2. Implement metrics gathering
Metrics are gathered by Geo::MetricsUpdateWorker
, persisted in GeoNodeStatus
for display in the UI, and sent to Prometheus:
-
Add the following fields to Geo Node Status example responses in doc/api/geo_nodes.md
:alert_management_alert_metric_images_count
alert_management_alert_metric_images_checksum_total_count
alert_management_alert_metric_images_checksummed_count
alert_management_alert_metric_images_checksum_failed_count
alert_management_alert_metric_images_synced_count
alert_management_alert_metric_images_failed_count
alert_management_alert_metric_images_registry_count
alert_management_alert_metric_images_verification_total_count
alert_management_alert_metric_images_verified_count
alert_management_alert_metric_images_verification_failed_count
alert_management_alert_metric_images_synced_in_percentage
alert_management_alert_metric_images_verified_in_percentage
-
Add the same fields to GET /geo_nodes/status
example response inee/spec/fixtures/api/schemas/public_api/v4/geo_node_status.json
. -
Add the following fields to the Sidekiq metrics
table indoc/administration/monitoring/prometheus/gitlab_metrics.md
:geo_alert_management_alert_metric_images
geo_alert_management_alert_metric_images_checksum_total
geo_alert_management_alert_metric_images_checksummed
geo_alert_management_alert_metric_images_checksum_failed
geo_alert_management_alert_metric_images_synced
geo_alert_management_alert_metric_images_failed
geo_alert_management_alert_metric_images_registry
geo_alert_management_alert_metric_images_verification_total
geo_alert_management_alert_metric_images_verified
geo_alert_management_alert_metric_images_verification_failed
Alert Metric Image replication and verification metrics should now be available in the API, the Admin > Geo > Nodes
view, and Prometheus.
Step 3. Implement the GraphQL API
The GraphQL API is used by Admin > Geo > Replication Details
views, and is directly queryable by administrators.
-
Add a new field to GeoNodeType
inee/app/graphql/types/geo/geo_node_type.rb
:field :alert_management_alert_metric_image_registries, ::Types::Geo::AlertManagement::MetricImageRegistryType.connection_type, null: true, resolver: ::Resolvers::Geo::AlertManagement::MetricImageRegistriesResolver, description: 'Find Alert Metric Image registries on this Geo node', feature_flag: :geo_alert_management_alert_metric_image_replication
-
Add the new alert_management_alert_metric_image_registries
field name to theexpected_fields
array inee/spec/graphql/types/geo/geo_node_type_spec.rb
. -
Create ee/app/graphql/resolvers/geo/alert_management_alert_metric_image_registries_resolver.rb
:# frozen_string_literal: true module Resolvers module Geo class AlertManagement::MetricImageRegistriesResolver < BaseResolver type ::Types::Geo::GeoNodeType.connection_type, null: true include RegistriesResolver end end end
-
Create ee/spec/graphql/resolvers/geo/alert_management_alert_metric_image_registries_resolver_spec.rb
:# frozen_string_literal: true require 'spec_helper' RSpec.describe Resolvers::Geo::AlertManagement::MetricImageRegistriesResolver do it_behaves_like 'a Geo registries resolver', :geo_alert_management_alert_metric_image_registry end
-
Create ee/app/finders/geo/alert_management_alert_metric_image_registry_finder.rb
:# frozen_string_literal: true module Geo class AlertManagement::MetricImageRegistryFinder include FrameworkRegistryFinder end end
-
Create ee/spec/finders/geo/alert_management_alert_metric_image_registry_finder_spec.rb
:# frozen_string_literal: true require 'spec_helper' RSpec.describe Geo::AlertManagement::MetricImageRegistryFinder do it_behaves_like 'a framework registry finder', :geo_alert_management_alert_metric_image_registry end
-
Create ee/app/graphql/types/geo/alert_management_alert_metric_image_registry_type.rb
:# frozen_string_literal: true module Types module Geo # rubocop:disable Graphql/AuthorizeTypes because it is included class AlertManagement::MetricImageRegistryType < BaseObject include ::Types::Geo::RegistryType graphql_name 'AlertManagement::MetricImageRegistry' description 'Represents the Geo replication and verification state of a alert_management_alert_metric_image' field :alert_management_alert_metric_image_id, GraphQL::ID_TYPE, null: false, description: 'ID of the Alert Metric Image' end end end
-
Create ee/spec/graphql/types/geo/alert_management_alert_metric_image_registry_type_spec.rb
:# frozen_string_literal: true require 'spec_helper' RSpec.describe GitlabSchema.types['AlertManagement::MetricImageRegistry'] do it_behaves_like 'a Geo registry type' it 'has the expected fields (other than those included in RegistryType)' do expected_fields = %i[alert_management_alert_metric_image_id] expect(described_class).to have_graphql_fields(*expected_fields).at_least end end
-
Add integration tests for providing AlertManagement::MetricImage registry data to the frontend via the GraphQL API, by duplicating and modifying the following shared examples in ee/spec/requests/api/graphql/geo/registries_spec.rb
:it_behaves_like 'gets registries for', { field_name: 'alertManagmentMetricImageRegistries', registry_class_name: 'AlertManagement::MetricImageRegistry', registry_factory: :geo_alert_management_alert_metric_image_registry, registry_foreign_key_field_name: 'alertManagmentMetricImageId' }
-
Update the GraphQL reference documentation: bundle exec rake gitlab:graphql:compile_docs
Individual Alert Metric Image replication and verification data should now be available via the GraphQL API.
Step 4. Handle batch destroy
If batch destroy logic is implemented for a replicable, then that logic must be "replicated" by Geo secondaries. The easiest way to do this is use Geo::BatchEventCreateWorker
to bulk insert a delete event for each replicable.
For example, if FastDestroyAll
is used, then you may be able to use begin_fast_destroy
and finalize_fast_destroy
hooks, like we did for uploads.
Or if a special service is used to batch delete records and their associated data, then you probably need to hook into that service, like we did for job artifacts.
As illustrated by the above two examples, batch destroy logic cannot be handled automatically by Geo secondaries without restricting the way other teams perform batch destroys. It is up to you to produce Geo::BatchEventCreateWorker
attributes before the records are deleted, and then enqueue Geo::BatchEventCreateWorker
after the records are deleted.
-
Ensure that any batch destroy of this replicable is replicated to secondary sites -
Regardless of implementation details, please verify in specs that when the parent object is removed, the new Geo::Event
records are created:
describe '#destroy' do
subject { create(:alert_management_alert_metric_image) }
context 'when running in a Geo primary node' do
let_it_be(:primary) { create(:geo_node, :primary) }
let_it_be(:secondary) { create(:geo_node) }
it 'logs an event to the Geo event log when bulk removal is used', :sidekiq_inline do
stub_current_geo_node(primary)
expect { subject.project.destroy! }.to change(Geo::Event.where(replicable_name: :alert_management_alert_metric_image, event_name: :deleted), :count).by(1)
payload = Geo::Event.where(replicable_name: :alert_management_alert_metric_image, event_name: :deleted).last.payload
expect(payload['model_record_id']).to eq(subject.id)
expect(payload['blob_path']).to eq(subject.relative_path)
expect(payload['uploader_class']).to eq('AlertManagement::MetricImageUploader')
end
end
end
Release Geo support of Alert Metric Images
-
In the rollout issue you created when creating the feature flag, modify the Roll Out Steps: -
Cross out any steps related to testing on production GitLab.com, because Geo is not running on production GitLab.com at the moment. -
Add a step to Test replication and verification of Alert Metric Images on a non-GDK-deployment. For example, using GitLab Environment Toolkit
. -
Add a step to Ping the Geo PM and EM to coordinate testing
. For example, you might add steps to generate Alert Metric Images, and then a Geo engineer may take it from there.
-
-
In ee/config/feature_flags/development/geo_alert_management_alert_metric_image_replication.yml
, setdefault_enabled: true
-
In ee/app/graphql/types/geo/geo_node_type.rb
, remove thefeature_flag
option for the released type:field :alert_management_alert_metric_image_registries, ::Types::Geo::AlertManagement::MetricImageRegistryType.connection_type, null: true, resolver: ::Resolvers::Geo::AlertManagement::MetricImageRegistriesResolver, description: 'Find Alert Metric Image registries on this Geo node', feature_flag: :geo_alert_management_alert_metric_image_replication # REMOVE THIS LINE
-
Run bundle exec rake gitlab:graphql:compile_docs
after the step above to regenerate the GraphQL docs. -
Add a row for Alert Metric Images to the Data types
table in Geo data types support -
Add a row for Alert Metric Images to the Limitations on replication/verification
table in Geo data types support. If the row already exists, then update it to show that Replication and Verification is released in the current version.