Skip to content

Elasticsearch indexing checkbox does not mark all migrations as completed

Summary

Note: Found while doing ruby 3 compatibility testing. However this issue is NOT caused by the upgrade to ruby 3

Using the Elasticsearch indexing checkbox in the Admin - Settings - Advanced Search UI does not mark all Advanced Search migrations as completed.

Steps to reproduce

  1. setup your gdk to use Advanced Search, you must have a new Elasticsearch or Open Search instance (cannot have any pre-existing indexes)
  2. check Elasticsearch indexing from the Admin - Settings - Advanced Search UI
  3. click Save
  4. click the Index all projects button
  5. Eventually, you will see a warning/alert stating Elasticsearch migration halted
  6. note the migrations index is empty (not expected):
curl --request GET \
  --url http://localhost:9200/gitlab-development-migrations/_search \
  --header 'Content-Type: application/json' \
  --data '{
        "query": {
                "bool": {
                        "must": [
                                {
                                        "match_all": {}
                                }
                        ],
                        "filter": [

                        ]
                }
        },
        "size": 1500
}' | jq

What is the current bug behavior?

Completely new users of Advanced Search will have a failed migration alert. It is the first migration (which is marked as obsolete) and should never be attempted to run. This migration is expected to exist and doesn't so we get an error in the logs and an alert (see more on the alert/error below)

What is the expected correct behavior?

When an instance is setup for Advanced Search for the first time, all migrations should be marked as completed (including obsolete migrations). This is what happens when the gitlab:elastic:index rake task is run. If you run that rake task, the migrations index correctly gets filled out with all migrations added to it and marked as completed

Relevant logs and/or screenshots

image

`elasticsearch.log` entries 
{"severity":"INFO","time":"2023-01-05T18:30:04.691Z","correlation_id":"b4990464a7e0aad1b53f9fc4d58b2d9a","message":"MigrationWorker: migration[ApplyMaxAnalyzedOffset] executing migrate method"}
{"severity":"INFO","time":"2023-01-05T18:30:04.691Z","correlation_id":"b4990464a7e0aad1b53f9fc4d58b2d9a","message":"[Elastic::Migration: 20201105181100] Migration has been deleted in the last major version upgrade.Migrations are supposed to be finished before upgrading major version https://docs.gitlab.com/ee/update/#upgrading-to-a-new-major-version .To correct this issue, recreate your index from scratch: https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch_troubleshooting.html#last-resort-to-recreate-an-index."}
{"severity":"INFO","time":"2023-01-05T18:30:04.691Z","correlation_id":"b4990464a7e0aad1b53f9fc4d58b2d9a","message":"[Elastic::Migration: 20201105181100] Halting migration with {}"}

Workaround

Two options to fix this - both require host access

  1. from the rails console, running ::Elastic::DataMigrationService.mark_all_as_completed!. This can only be done if no upgrades to GitLab have happened. New migrations could have been introduced during an upgrade.
  2. from the command line, reindexing from scratch using the rake task: gitlab:elastic:index

Possible fixes

EE::ApplicationSettings::UpdateService#find_or_create_elasticsearch_index is called when the checkbox is checked and the form is saved. ::Elastic::DataMigrationService.mark_all_as_completed! needs to be used to mark all migrations as completed (as is done in the rake task) after the migrations index is created. I believe the migrations should only be marked as completed if the migrations index is newly created. We should test to confirm the behavior is as expected in the case where:

  • a migrations index already exists
  • there are pending migrations
  • the checkbox is unchecked (and saved)
  • the checkbox is checked (and saved)
Edited by Terri Chu