gitaly fails to start
When gitaly-wrapper is started, it will parse the pid file. If the pid file is empty, gitaly-wrapper will panic due to invalid json syntax.
Step to reproduce:
- Stop gitaly,
gitlab-ctl stop gitaly
- Modify pid file and delete the pid number.
gitlab-ctl start gitaly
- gitaly will no longer be able to start until the pid file is modified or deleted.
Errors will loop with this:
{"level":"info","msg":"finding gitaly","pid_file":"/var/opt/gitlab/gitaly/gitaly.pid","time":"2020-02-11T09:05:38Z","wrapper":287}
{"error":"strconv.Atoi: parsing \"\": invalid syntax","level":"fatal","msg":"find gitaly","time":"2020-02-11T09:05:38Z","wrapper":287}
When accessing the /projects folder errors will look like this:
GRPC::Unavailable (14:failed to connect to all addresses):
lib/gitlab/gitaly_client.rb:168:in `execute'
lib/gitlab/gitaly_client.rb:146:in `block in call'
lib/gitlab/gitaly_client.rb:174:in `measure_timings'
lib/gitlab/gitaly_client.rb:145:in `call'
lib/gitlab/gitaly_client/server_service.rb:14:in `info'
lib/gitlab/gitaly_client.rb:435:in `filesystem_id'
lib/gitlab/gitaly_client.rb:425:in `can_use_disk?'
lib/gitlab/git/rugged_impl/use_rugged.rb:15:in `use_rugged?'
lib/gitlab/git/rugged_impl/commit.rb:38:in `find_commit'
lib/gitlab/git/commit.rb:66:in `find'
lib/gitlab/metrics/instrumentation.rb:161:in `block in find'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `find'
app/models/repository.rb:1140:in `find_commit'
lib/gitlab/metrics/instrumentation.rb:161:in `block in find_commit'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `find_commit'
app/models/repository.rb:114:in `commit'
lib/gitlab/metrics/instrumentation.rb:161:in `block in commit'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `commit'
app/models/project.rb:786:in `commit'
lib/gitlab/metrics/instrumentation.rb:161:in `block in commit'
lib/gitlab/metrics/method_call.rb:36:in `measure'
lib/gitlab/metrics/instrumentation.rb:161:in `commit'
lib/gitlab/cache/ci/project_pipeline_status.rb:118:in `block in commit'
lib/gitlab/utils/strong_memoize.rb:30:in `strong_memoize'
lib/gitlab/cache/ci/project_pipeline_status.rb:117:in `commit'
lib/gitlab/cache/ci/project_pipeline_status.rb:52:in `load_status'
lib/gitlab/cache/ci/project_pipeline_status.rb:23:in `block in load_in_batch_for_projects'
lib/gitlab/cache/ci/project_pipeline_status.rb:21:in `load_in_batch_for_projects'
app/helpers/projects_helper.rb:211:in `load_pipeline_status'
app/views/shared/projects/_list.html.haml:36:in `_app_views_shared_projects__list_html_haml__3922925748592422687_70241759975120'
app/views/dashboard/projects/_projects.html.haml:1:in `_app_views_dashboard_projects__projects_html_haml__4205044176718215879_70241759910620'
app/views/dashboard/projects/index.html.haml:15:in `_app_views_dashboard_projects_index_html_haml__2243562159645393040_70241757579200'
app/controllers/application_controller.rb:121:in `render'
app/controllers/dashboard/projects_controller.rb:60:in `block in render_projects'
lib/gitlab/gitaly_client.rb:314:in `allow_n_plus_1_calls'
app/controllers/dashboard/projects_controller.rb:59:in `render_projects'
app/controllers/dashboard/projects_controller.rb:19:in `block (2 levels) in index'
app/controllers/dashboard/projects_controller.rb:17:in `index'
lib/gitlab/session.rb:11:in `with_session'
app/controllers/application_controller.rb:467:in `set_session_storage'
lib/gitlab/i18n.rb:55:in `with_locale'
lib/gitlab/i18n.rb:61:in `with_user_locale'
app/controllers/application_controller.rb:461:in `set_locale'
lib/gitlab/application_context.rb:18:in `with_context'
app/controllers/application_controller.rb:453:in `set_current_context'
lib/gitlab/error_tracking.rb:34:in `with_context'
app/controllers/application_controller.rb:545:in `sentry_context'
lib/gitlab/middleware/rails_queue_duration.rb:27:in `call'
lib/gitlab/metrics/rack_middleware.rb:17:in `block in call'
lib/gitlab/metrics/transaction.rb:62:in `run'
lib/gitlab/metrics/rack_middleware.rb:17:in `call'
lib/gitlab/request_profiler/middleware.rb:17:in `call'
lib/gitlab/middleware/go.rb:20:in `call'
lib/gitlab/etag_caching/middleware.rb:13:in `call'
lib/gitlab/middleware/multipart.rb:117:in `call'
lib/gitlab/middleware/read_only/controller.rb:52:in `call'
lib/gitlab/middleware/read_only.rb:18:in `call'
lib/gitlab/middleware/basic_health_check.rb:25:in `call'
lib/gitlab/middleware/request_context.rb:23:in `call'
config/initializers/fix_local_cache_middleware.rb:9:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:49:in `call'
lib/gitlab/middleware/release_env.rb:12:in `call'
Completed 503 Service Unavailable in 783ms (Views: 8.1ms | ActiveRecord: 99.7ms | Elasticsearch: 0.0ms)