Skip to content

Add a migration to reindex already existing group wikis

Ravi Kumar requested to merge 336100-index-already-existing-groupwiki into master

What does this MR do and why?

We have already implemented the indexing of the GroupWikis but we still need a migration to reindex all the GroupWikis which were created or updated before the implementation of the GroupWikis. This migration will reindex all the GroupWikis which are enabled to use elasticsearch

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Make sure Elasticsearch is enabled

  1. Create a group wiki in the GDK

  2. To simulate the scenario of not having group wikis in Elasticsearch, you have to delete all the group wikis from the elastic

curl -H 'Content-type: application/json' -XPOST 'http://localhost:9200/gitlab-development-wikis/_delete_by_query -d'
{
  "body": {
    "query": {
      "regexp": {
        "rid": "wiki_group_[0-9].*"
      }
    }
  }
}'
  1. Open the Rails console
bundle exec rails c
  1. Run the following command
require_relative 'ee/elastic/migrate/20230625500000_backfill_existing_group_wiki.rb' 
BackfillExistingGroupWiki.new(20230625500000).migrate
  1. Check group wikis on ES
curl -H 'Content-type: application/json' -XGET 'http://localhost:9200/gitlab-development-wikis/_search -d'
{
  "body": {
    "query": {
      "regexp": {
        "rid": "wiki_group_[0-9].*"
      }
    }
  }
}'

It should not be empty

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Approximate time to completion

~ 24 hours

No. of iterations needed

56191/200 =~ 280

[1] pry(main)> number_of_groups = GroupWikiRepository.count
=> 56191
[2] pry(main)> throttle_delay = 5.minute
=> 5 minutes
[3] pry(main)> batch_size = 200
=> 200
[4] pry(main)> ((number_of_groups / batch_size) * throttle_delay) / 1.hour
=> 23

Related to #336100 (closed)

Edited by Ravi Kumar

Merge request reports

Loading