Fail to deploy chart on AKS with domain hosted on AWS Route53 ( Sidekiq, unicorn and gitlab-runner fail)
Summary
Sidekiq, unicorn and gitlab-runner fail. I'm deploying on Azure AKS, but use AWS Route53 for the domain. I don't have much experience with all of this.
Steps to reproduce
I deployed the chart on Azure. Here are the steps I took:
RESOURCE_GROUP=gitlab_k8s
CLUSTER_NAME=gitlab-k8s-cluster
GITLAB_IP_NAME=myGitlabPublicIP
az group create --name $RESOURCE_GROUP --location eastus
az aks create --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --node-count 2 --enable-addons monitoring --generate-ssh-keys
az aks get-credentials --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME
kubectl create -f rbac-config.yaml
helm init --service-account tiller
helm repo add gitlab https://charts.gitlab.io/
helm repo update
# create ip inside noderesource group
NODE_RESOURCE_GROUP=$(az aks show --resource-group $RESOURCE_GROUP --name $CLUSTER_NAME --query nodeResourceGroup -o tsv)
az network public-ip create --resource-group $NODE_RESOURCE_GROUP --name $GITLAB_IP_NAME --allocation-method static
PUBLIC_IP=$(az network public-ip show --resource-group $NODE_RESOURCE_GROUP --name $GITLAB_IP_NAME --query ipAddress --output tsv)
# add public ip to AWS route53
# database
POSTGRES_PASSWORD=<clear txt for dev>
POSTGRES_HOST=XXXX
POSTGRES_ADMIN=pgadmin@$POSTGRES_HOST
az postgres server create \
--resource-group $RESOURCE_GROUP \
--name $POSTGRES_HOST \
--location eastus \
--admin-user pgadmin \
--admin-password $POSTGRES_PASSWORD \
--sku-name GP_Gen5_2 \
--version 9.6
# activate pg_trgm extension
# allow traffic from local IP to DB to run the commands below
psql --host=$POSTGRES_HOST.postgres.database.azure.com --username=pgadmin@$POSTGRES_HOST postgres
-> CREATE DATABASE gitlabhq_production OWNER pgadmin;
# log into gitlabhq_production database
psql --host=$POSTGRES_HOST.postgres.database.azure.com --username=pgadmin@$POSTGRES_HOST gitlabhq_production
-> CREATE EXTENSION pg_trgm;
#-> CREATE USER gitlab WITH CREATEDB ENCRYPTED PASSWORD '.~BgFxhxkQEu';
#-> grant all privileges on database postgres to gitlab ;
# create k8s secret for postgres
kubectl create secret generic postgres-password --from-literal=password=$POSTGRES_PASSWORD
# redis
az redis create \
--location eastus \
--name XXX_GITLAB_REDIS \
--resource-group $RESOURCE_GROUP\
--sku Basic\
--vm-size C0
# allow redis access on port 6379
REDIS_ACCESS_KEY=$(az redis list-keys --name XXX_GITLAB_REDIS --resource-group $RESOURCE_GROUP -o json | jq --raw-output '.primaryKey')
REDIS_HOST=$(az redis show --name XXX_GITLAB_REDIS --resource-group $RESOURCE_GROUP -o json | jq --raw-output '.hostName')
# create k8s secret for redis
kubectl create secret generic redis-access-key --from-literal=key=$REDIS_ACCESS_KEY
# email configuration
kubectl create secret generic email-password --from-env-file=email_password.env
helm upgrade --install gitlab gitlab/gitlab \
--timeout 600 \
--set global.hosts.domain=ourdomain.com \
--set global.hosts.externalIP=$PUBLIC_IP \
--set certmanager-issuer.email=johannes@ourdomain.com \
--set postgresql.install=false \
--set global.psql.host=XXXX.postgres.database.azure.com \
--set global.psql.password.secret=postgres-password \
--set global.psql.password.key=password \
--set global.psql.username=$POSTGRES_ADMIN \
--set global.smtp.enabled=true \
--set global.smtp.address=smtp.gmail.com \
--set global.smtp.port=587 \
--set global.smtp.openssl_verify_mode=ssl/tls \
--set global.smtp.user_name=gitlab@ourdomain.com \
--set global.smtp.password.secret=email-password \
--set global.smtp.password.key=password \
--set global.smtp.authentication=login \
--set global.email.from=gitlab@ourdomain.com \
--set global.email.display_name="..." \
--set global.email.reply_to=gitlab@ourdomain.com \
--set global.email.subject_suffix="..." \
--set redis.enabled=false \
--set global.redis.host=$REDIS_HOST \
--set global.redis.password.secret=redis-access-key \
--set global.redis.password.key=key \
--set global.redis.port=6379
# remove:
# helm delete gitlab ; helm del --purge gitlab
# az group delete --name $RESOURCE_GROUP --yes --no-wait
I added the following A records to route 53: gitlab.ourdomain.com and *.gitlab.ourdomain.com, both pointing to the same PUBLIC_IP set in the script above.
Configuration used
I don't have a yaml configuration? See above for the steps I took
Current behavior
Here is the output from kubectl get pods:
NAME READY STATUS RESTARTS AGE
cm-acme-http-solver-lsgfv 1/1 Running 0 44m
cm-acme-http-solver-ptmng 1/1 Running 0 44m
gitlab-certmanager-664f847794-pptk6 1/1 Running 0 89m
gitlab-gitaly-0 1/1 Running 0 50m
gitlab-gitlab-runner-5d4bf868d-7p4nr 0/1 CrashLoopBackOff 20 89m
gitlab-gitlab-shell-5fcf6f8685-dd5cv 1/1 Running 0 51m
gitlab-gitlab-shell-5fcf6f8685-w6j48 1/1 Running 0 51m
gitlab-issuer.2-gpl6g 0/1 Completed 0 51m
gitlab-migrations.2-szk46 0/1 Completed 0 51m
gitlab-minio-8dc7f5964-tq6sv 1/1 Running 0 89m
gitlab-minio-create-buckets.2-9mgcb 0/1 Completed 0 51m
gitlab-nginx-ingress-controller-688b48c456-b947v 1/1 Running 0 89m
gitlab-nginx-ingress-controller-688b48c456-spwbk 1/1 Running 0 89m
gitlab-nginx-ingress-controller-688b48c456-xrrvf 1/1 Running 0 89m
gitlab-nginx-ingress-default-backend-cb9857f68-4jzjg 1/1 Running 0 89m
gitlab-nginx-ingress-default-backend-cb9857f68-749jc 1/1 Running 0 89m
gitlab-prometheus-server-8cf4fdd8-tj75t 2/2 Running 0 89m
gitlab-registry-5779d776d6-pzbbg 1/1 Running 0 88m
gitlab-registry-5779d776d6-tv9rw 1/1 Running 0 89m
gitlab-sidekiq-all-in-1-764fd4d6fb-c9npk 1/1 Running 17 51m
gitlab-task-runner-57459b48cf-7v47k 1/1 Running 0 51m
gitlab-unicorn-7cdbc955d6-sd7lz 1/2 CrashLoopBackOff 14 51m
gitlab-unicorn-8678d476b8-hxd67 1/2 CrashLoopBackOff 18 89m
gitlab-unicorn-8678d476b8-jf685 1/2 CrashLoopBackOff 18 88m
Services:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
cm-acme-http-solver-6jrs6 NodePort 10.0.236.58 <none> 8089:31685/TCP 53m
cm-acme-http-solver-mxpwq NodePort 10.0.22.225 <none> 8089:30926/TCP 53m
gitlab-gitaly ClusterIP None <none> 8075/TCP,9236/TCP 98m
gitlab-gitlab-shell ClusterIP 10.0.44.243 <none> 22/TCP 98m
gitlab-minio-svc ClusterIP 10.0.13.125 <none> 9000/TCP 98m
gitlab-nginx-ingress-controller LoadBalancer 10.0.195.21 PUBLIC_IP 80:30825/TCP,443:32260/TCP,22:32430/TCP 98m
gitlab-nginx-ingress-controller-metrics ClusterIP 10.0.15.154 <none> 9913/TCP 98m
gitlab-nginx-ingress-controller-stats ClusterIP 10.0.170.141 <none> 18080/TCP 98m
gitlab-nginx-ingress-default-backend ClusterIP 10.0.115.132 <none> 80/TCP 98m
gitlab-prometheus-server ClusterIP 10.0.59.167 <none> 80/TCP 98m
gitlab-registry ClusterIP 10.0.75.36 <none> 5000/TCP 98m
gitlab-unicorn ClusterIP 10.0.114.241 <none> 8080/TCP,8181/TCP 98m
kubernetes ClusterIP 10.0.0.1 <none> 443/TCP 126m
gitlab-runner, sidekiq and unicorn don't stabilize. My guess is that unicorn is the culprit (in conjunction with the domain?). Output for kubectl logs gitlab-unicorn-7cdbc955d6-sd7lz -c unicorn:
> kubectl logs gitlab-unicorn-7cdbc955d6-sd7lz -c unicorn
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/resque.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
Copying unicorn.rb into /srv/gitlab/config
+ exec /bin/sh -c /scripts/process-wrapper
Starting unicorn
==> /var/log/gitlab/api_json.log <==
# Logfile created on 2018-11-28 17:30:31 +0000 by logger.rb/56815
==> /var/log/gitlab/grpc.log <==
# Logfile created on 2018-11-28 17:30:31 +0000 by logger.rb/56815
==> /var/log/gitlab/production.log <==
==> /var/log/gitlab/production_json.log <==
# Logfile created on 2018-11-28 17:30:31 +0000 by logger.rb/56815
I, [2018-11-29T11:05:22.726856 #10] INFO -- : Refreshing Gem list
WARNING: This version of GitLab depends on gitlab-shell 8.4.1, but you're running Unknown. Please update gitlab-shell.
bundler: failed to load command: unicorn (/srv/gitlab/vendor/bundle/ruby/2.4.0/bin/unicorn)
NoMethodError: undefined method `/' for :ssl:Symbol
/srv/gitlab/config/initializers/smtp_settings.rb:12:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:652:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:651:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/lib/ruby/2.4.0/tsort.rb:228:in `block in tsort_each'
/usr/lib/ruby/2.4.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/lib/ruby/2.4.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/usr/lib/ruby/2.4.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.4.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `tsort_each_child'
/usr/lib/ruby/2.4.0/tsort.rb:415:in `call'
/usr/lib/ruby/2.4.0/tsort.rb:415:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.4.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/lib/ruby/2.4.0/tsort.rb:347:in `each'
/usr/lib/ruby/2.4.0/tsort.rb:347:in `call'
/usr/lib/ruby/2.4.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/lib/ruby/2.4.0/tsort.rb:226:in `tsort_each'
/usr/lib/ruby/2.4.0/tsort.rb:205:in `tsort_each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
/srv/gitlab/config/environment.rb:11:in `<top (required)>'
/srv/gitlab/config.ru:19:in `require'
/srv/gitlab/config.ru:19:in `block in <main>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/rack-1.6.10/lib/rack/builder.rb:55:in `instance_eval'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/rack-1.6.10/lib/rack/builder.rb:55:in `initialize'
/srv/gitlab/config.ru:1:in `new'
/srv/gitlab/config.ru:1:in `<main>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn.rb:56:in `eval'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn.rb:56:in `block in builder'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:752:in `build_app!'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/lib/unicorn/http_server.rb:129:in `start'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/unicorn-5.1.0/bin/unicorn:126:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/bin/unicorn:23:in `<top (required)>'
Expected behavior
Everything should deploy just fine and I can access my gitlab at gitlab.ourdomain.com
Versions
- Chart: Whatever is installed with helm upgrade --install gitlab gitlab/gitlab
- Platform:
- Cloud: AKS
- Kubernetes: (
kubectl version
)- Client: version.Info{Major:"1", Minor:"12", GitVersion:"v1.12.2", GitCommit:"17c77c7898218073f14c8d573582e8d2313dc740", GitTreeState:"clean", BuildDate:"2018-10-24T06:54:59Z", GoVersion:"go1.10.4", Compiler:"gc", Platform:"linux/amd64"}
- Server: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.11", GitCommit:"1bfeeb6f212135a22dc787b73e1980e5bccef13d", GitTreeState:"clean", BuildDate:"2018-09-28T21:35:22Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
- Helm: (
helm version
)- Client:Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
- Server:Server: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Relevant logs
See above. Some more logs: Sidekiq:
⇒ kubectl logs gitlab-sidekiq-all-in-1-764fd4d6fb-c9npk
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/resque.yml
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/sidekiq_queues.yml
Copying other config files found in /var/opt/gitlab/templates
Copying smtp_settings.rb into /srv/gitlab/config
+ exec /bin/sh -c /scripts/process-wrapper
Starting Sidekiq
==> /var/log/gitlab/api_json.log <==
# Logfile created on 2018-11-28 17:30:31 +0000 by logger.rb/56815
==> /var/log/gitlab/grpc.log <==
# Logfile created on 2018-11-28 17:30:31 +0000 by logger.rb/56815
==> /var/log/gitlab/production.log <==
==> /var/log/gitlab/production_json.log <==
# Logfile created on 2018-11-28 17:30:31 +0000 by logger.rb/56815
==> /var/log/gitlab/repocheck.log <==
WARNING: This version of GitLab depends on gitlab-shell 8.4.1, but you're running Unknown. Please update gitlab-shell.
2018-11-29T11:14:59.735Z 11 TID-gt1qakfv7 INFO: Booting Sidekiq 5.2.1 with redis options {:host=>"me-gitlab-redis.redis.cache.windows.net", :port=>6379, :password=>"REDACTED", :namespace=>"resque:gitlab", :id=>"Sidekiq-server-PID-11", :url=>nil}
2018-11-29T11:14:59.861Z 11 TID-gt1qakfv7 INFO: Cron Jobs - deleted job with name: update_all_mirrors_worker
2018-11-29T11:14:59.868Z 11 TID-gt1qakfv7 INFO: Cron Jobs - deleted job with name: geo_sidekiq_cron_config_worker
2018-11-29T11:14:59.873Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: stuck_ci_jobs_worker
2018-11-29T11:14:59.882Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: pipeline_schedule_worker
2018-11-29T11:14:59.887Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: expire_build_artifacts_worker
2018-11-29T11:14:59.891Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: repository_check_worker
2018-11-29T11:14:59.895Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: admin_email_worker
2018-11-29T11:14:59.900Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: repository_archive_cache_worker
2018-11-29T11:14:59.907Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: historical_data_worker
2018-11-29T11:14:59.912Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: ldap_sync_worker
2018-11-29T11:14:59.917Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: ldap_group_sync_worker
2018-11-29T11:14:59.922Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_metrics_update_worker
2018-11-29T11:14:59.927Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_repository_sync_worker
2018-11-29T11:14:59.930Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_file_download_dispatch_worker
2018-11-29T11:14:59.934Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_prune_event_log_worker
2018-11-29T11:14:59.939Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_repository_verification_primary_batch_worker
2018-11-29T11:14:59.945Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_repository_verification_secondary_scheduler_worker
2018-11-29T11:14:59.953Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_migrated_local_files_clean_up_worker
2018-11-29T11:14:59.958Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: import_export_project_cleanup_worker
2018-11-29T11:14:59.962Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: ci_archive_traces_cron_worker
2018-11-29T11:14:59.966Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: requests_profiles_worker
2018-11-29T11:14:59.971Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: remove_expired_members_worker
2018-11-29T11:14:59.978Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: remove_expired_group_links_worker
2018-11-29T11:14:59.984Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: prune_old_events_worker
2018-11-29T11:14:59.989Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: trending_projects_worker
2018-11-29T11:14:59.994Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: remove_unreferenced_lfs_objects_worker
2018-11-29T11:14:59.998Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: stuck_import_jobs_worker
2018-11-29T11:15:00.004Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: gitlab_usage_ping_worker
2018-11-29T11:15:00.017Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: pseudonymizer_worker
2018-11-29T11:15:00.022Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: clear_shared_runners_minutes_worker
2018-11-29T11:15:00.031Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: remove_old_web_hook_logs_worker
2018-11-29T11:15:00.035Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: stuck_merge_jobs_worker
2018-11-29T11:15:00.039Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: pages_domain_verification_cron_worker
2018-11-29T11:15:00.045Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: issue_due_scheduler_worker
2018-11-29T11:15:00.049Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: prune_web_hook_logs_worker
2018-11-29T11:15:00.059Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: update_all_mirrors_worker
2018-11-29T11:15:00.064Z 11 TID-gt1qakfv7 INFO: Cron Jobs - add job with name: geo_sidekiq_cron_config_worker
undefined method `/' for :ssl:Symbol
/srv/gitlab/config/initializers/smtp_settings.rb:12:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `block in load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:268:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:652:in `block in load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/notifications.rb:166:in `instrument'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:651:in `load_config_initializer'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:616:in `block (2 levels) in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/engine.rb:615:in `block in <class:Engine>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:30:in `run'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/lib/ruby/2.4.0/tsort.rb:228:in `block in tsort_each'
/usr/lib/ruby/2.4.0/tsort.rb:350:in `block (2 levels) in each_strongly_connected_component'
/usr/lib/ruby/2.4.0/tsort.rb:422:in `block (2 levels) in each_strongly_connected_component_from'
/usr/lib/ruby/2.4.0/tsort.rb:431:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.4.0/tsort.rb:421:in `block in each_strongly_connected_component_from'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:44:in `tsort_each_child'
/usr/lib/ruby/2.4.0/tsort.rb:415:in `call'
/usr/lib/ruby/2.4.0/tsort.rb:415:in `each_strongly_connected_component_from'
/usr/lib/ruby/2.4.0/tsort.rb:349:in `block in each_strongly_connected_component'
/usr/lib/ruby/2.4.0/tsort.rb:347:in `each'
/usr/lib/ruby/2.4.0/tsort.rb:347:in `call'
/usr/lib/ruby/2.4.0/tsort.rb:347:in `each_strongly_connected_component'
/usr/lib/ruby/2.4.0/tsort.rb:226:in `tsort_each'
/usr/lib/ruby/2.4.0/tsort.rb:205:in `tsort_each'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/initializable.rb:54:in `run_initializers'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/railties-4.2.10/lib/rails/application.rb:352:in `initialize!'
/srv/gitlab/config/environment.rb:11:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `block in require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:240:in `load_dependency'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/activesupport-4.2.10/lib/active_support/dependencies.rb:274:in `require'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/cli.rb:275:in `boot_system'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.1/lib/sidekiq/cli.rb:53:in `run'
/srv/gitlab/vendor/bundle/ruby/2.4.0/gems/sidekiq-5.2.1/bin/sidekiq:12:in `<top (required)>'
/srv/gitlab/vendor/bundle/ruby/2.4.0/bin/sidekiq:23:in `load'
/srv/gitlab/vendor/bundle/ruby/2.4.0/bin/sidekiq:23:in `<top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `load'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `kernel_load'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:28:in `run'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli.rb:424:in `exec'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/usr/lib/ruby/gems/2.4.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>'
/srv/gitlab/bin/bundle:3:in `load'
/srv/gitlab/bin/bundle:3:in `<main>'
gitlab-runner:
kubectl logs gitlab-gitlab-runner-5d4bf868d-7p4nr
WARNING: Running in user-mode.
WARNING: The user-mode requires you to manually start builds processing:
WARNING: $ gitlab-runner run
WARNING: Use sudo for system-mode:
WARNING: $ sudo gitlab-runner...
ERROR: Registering runner... failed runner=wO8cuGSA status=503 Service Temporarily Unavailable
PANIC: Failed to register this runner. Perhaps you are having network problems
Another unicorn instance shows another log:
⇒ kubectl logs gitlab-unicorn-8678d476b8-hxd67 -c unicorn
container_linux.go:247: starting container process caused "process_linux.go:359: container init caused \"rootfs_linux.go:54: mounting \\\"/var/lib/kubelet/pods/c0a50e7f-f3ba-11e8-b26d-d2b98067ba26/volume-subpaths/unicorn-config/unicorn/5\\\" to rootfs \\\"/var/lib/docker/overlay2/7772a024161d078283052a404e67242320a75899e5ed9866f81af789a9059b78/merged\\\" at \\\"/var/lib/docker/overlay2/7772a024161d078283052a404e67242320a75899e5ed9866f81af789a9059b78/merged/srv/gitlab/INSTALLATION_TYPE\\\" caused \\\"no such file or directory\\\"\""