Skip to content

Add schema_version to main index mapping

Terri Chu requested to merge 214601-add-schema_version-to-main-index into master

What does this MR do and why?

Related to #214601 (closed)

Adding schema_version to index mapping for the main index (contains, project, wiki_blobs, and blob document types).

Note: We would like to have a schema_version field available for future migrations in all document types.

This is prep work to be able to easily backfill traversal_ids for projects which will be needed for the IndexIntegrityWorker. The IndexIntegrityWorker (dev in progress) will need to pull all projects for a root namespace and I need the traversal_ids field populated for that. Currently it's only being populated for blobs and wiki_blobs.

Screenshots or screen recordings

N/A

How to set up and validate locally

prerequisites

setup Elasticsearch in gdk

check config.rb changes

  1. create an index from scratch: bundle exec rake gitlab:elastic:index
  2. verify the new mapping is there and has data (tests the config.rb changes) GET http://localhost:9200/gitlab-development/_search
{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"type": {
							"value": "project"
						}
					}
				}
			]
		}
	}
}

migration changes

  1. checkout the master branch
  2. recreate the indexes from scratch : bundle exec rake gitlab:elastic:index
  3. add a new project
  4. verify that it doesn't throw an error and indexes properly (without the new field)
  5. stop rails-background-jobs to prevent it from running the migration automatically: gdk stop rails-background-jobs
  6. start rails console
  7. run the migration in the rails console
  8. verify that the schema_version field exists in the mappings: GET http://localhost:9200/gitlab-development/_mappings
  9. add a new or update an existing project and run the indexing manually: Elastic::ProcessBookkeepingService.new.execute
  10. verify that the schema_version field exists and is populated for the project in the search response GET http://localhost:9200/gitlab-development/_search
{
	"query": {
		"bool": {
			"must": [
				{
					"term": {
						"type": {
							"value": "project"
						}
					}
				}
			]
		}
	}
}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Terri Chu

Merge request reports

Loading