Skip to content

Marks migrations as complete when creating a new index

Madelein van Niekerk requested to merge 387237-mark-migrations-complete into master

What does this MR do and why?

Marks Elastic migrations as complete when creating indices for the first time by enabling indexing on the UI.

Why

When creating indices using the rake task gitlab:elastic:index, the migrations index is created and all migrations are marked as complete. This is because the index is created with the correct content, mappings and settings as if all migrations have been run.

When creating the indices by enabling indexing from the Admin > Settings > Advanced Search UI by checking the Elasticsearch indexing box:

the migrations index is created but is empty. This will attempt to execute the migrations and will eventually lead to all migrations being halted.

Logs 
[Elastic::MigrationRecord]: Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"_index\":\"gitlab-development-migrations\",\"_id\":\"20201105181100\",\"found\":false}
MigrationWorker: migration[ApplyMaxAnalyzedOffset] executing migrate method
[Elastic::Migration: 20201105181100] Migration has been deleted in the last major version upgrade.Migrations are supposed to be finished before upgrading major version https://docs.gitlab.com/ee/update/#upgrading-to-a-new-major-version .To correct this issue, recreate your index from scratch: https://docs.gitlab.com/ee/integration/advanced_search/elasticsearch_troubleshooting.html#last-resort-to-recreate-an-index.
[Elastic::Migration: 20201105181100] Halting migration with {}
[Elastic::MigrationRecord]: Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"_index\":\"gitlab-development-migrations\",\"_id\":\"20201105181100\",\"found\":false}
[Elastic::MigrationRecord]: Elasticsearch::Transport::Transport::Errors::NotFound: [404] {\"_index\":\"gitlab-development-migrations\",\"_id\":\"20201105181100\",\"found\":false}
MigrationWorker: migration[ApplyMaxAnalyzedOffset] updating with completed: false
MigrationWorker: migration[ApplyMaxAnalyzedOffset] has been halted. All future migrations will be halted because of that. Exiting

See Steps to reproduce.

How

After updating the elasticsearch settings, we call find_or_create_elasticsearch_index which will create all necessary indices if they don't exist. If the migrations index does not exist, we create it and then mark all the migrations as complete using ::Elastic::DataMigrationService.mark_all_as_completed!.

How to set up and validate locally

  1. If elastic indexing is enabled, uncheck the Elasticsearch indexing checkbox on /admin/application_settings/advanced_search and save.
  2. Delete any existing development indices (gitlab-development*)
  3. Check the Elasticsearch indexing checkbox on /admin/application_settings/advanced_search and save. This will create new indices.
  4. Verify that there are 30 migration records and all have complete as true: 
    curl http://localhost:9200/gitlab-development-migrations/_search  --data '{"aggs": {"completed": {"terms": {"field": "completed"}}},"size": 0}' --header "Content-Type: application/json" | jq '.aggregations | .completed | .buckets'
  5. [Optional] Execute the migration worker rails runner "Elastic::MigrationWorker.new.perform" and verify that the only migration log is MigrationWorker: no migration available

[Optional] Test that if there is a migration index with a pending migration that nothing changes in the state of the migration index:

  1. Update one of the migration records to be not completed: 
    curl -X PUT http://localhost:9200/gitlab-development-migrations/_doc/20201116142400 --data '{"completed": false}' --header "Content-Type: application/json"
  2. Uncheck the Elasticsearch indexing checkbox on /admin/application_settings/advanced_search and save.
  3. Check the Elasticsearch indexing checkbox on /admin/application_settings/advanced_search and save.
  4. Verify that the migration record is still left as completed: false: 
    curl http://localhost:9200/gitlab-development-migrations/_doc/20201116142400 | jq '."_source"'

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #387237 (closed)

Edited by Madelein van Niekerk

Merge request reports