Reorder gitlab:elastic:index rake tasks to ensure wikis and database are completed even if projects error out

Summary

Initial indexing of wikis and the database can be missed when setting up elasticsearch due to the indexing of projects failing. The focus then becomes the error on a particular index, once this is resolved then a support ticket is created to investigate why search isn't showing more results. Often this is resolved by running the rake tasks for wikis and index manually.

Rake task gitlab-rake gitlab:elastic:index does

      Rake::Task["gitlab:elastic:create_empty_index"].invoke
      Rake::Task["gitlab:elastic:clear_index_status"].invoke
      Rake::Task["gitlab:elastic:index_repositories"].invoke
      Rake::Task["gitlab:elastic:index_wikis"].invoke
      Rake::Task["gitlab:elastic:index_database"].invoke

This issue is to investigate swapping the order so that gitlab:elastic:index_repositories is done last.

Steps to reproduce

Add the Linux Kernel Source to your GitLab instance to generate an indexing error. Configure elasticsearch Run rake task gitlab-rake gitlab:elastic:index

Example Project

What is the current bug behavior?

Rake task error's out without completing wikis and database causing confusion when checking elasticsearch status.

What is the expected correct behavior?

Index wikis and database before repositories.

Relevant logs and/or screenshots

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 16.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.4.5p335
Gem Version:    2.7.6
Bundler Version:1.16.6
Rake Version:   12.3.1
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.1
Go Version:     unknown
GitLab information
Version:        11.5.3-ee
Revision:       588b25b
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.8
URL:            https://astrachan-latest.asgks.net
HTTP Clone URL: https://astrachan-latest.asgks.net/some-group/some-project.git
SSH Clone URL:  git@astrachan-latest.asgks.net:some-group/some-project.git
Elasticsearch:  yes
Geo:            yes
Geo node:       Primary
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: saml, bitbucket
GitLab Shell
Version:        8.4.1
Repository storage paths:

• default: /var/opt/gitlab/git-data/repositories
• storage1: /srv/gitlab/storage1/repositories/repositories
• storage2: /srv/gitlab/storage2/repositories/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...
GitLab Shell version >= 8.4.1 ? ... OK (8.4.1)
hooks directories in repos are links: ...
1/13 ... ok
...
71/299 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
default ... OK
storage1 ... OK
storage2 ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/13 ... yes
..
71/299 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.5)
Git version >= 2.9.5 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 22
Elasticsearch version 5.1 - 5.5? ... no (5.6.8)
For more information see:
doc/integration/elasticsearch.md
Checking GitLab ... Finished

Possible fixes

Reorder rake tasks to have repositories as last.