Error importing repository https://gitlab.com/gitlab-org/gitlab-test.git into gitlab-org/gitlab-test - 13:CreateRepositoryFromURL: clone cmd wait: error executing git hook
Overview
Am seeing the following error in a number of GDK verify:*
CI jobs:
rake aborted!
Error importing repository https://gitlab.com/gitlab-org/gitlab-test.git into gitlab-org/gitlab-test - 13:CreateRepositoryFromURL: clone cmd wait: error executing git hook
error executing git hook
fatal: ref updates aborted by hook
: exit status 128.
/builds/gitlab-org/gitlab-development-kit/gitlab-development-kit/gitlab/app/workers/repository_import_worker.rb:35:in `perform'
/builds/gitlab-org/gitlab-development-kit/gitlab-development-kit/gitlab/ee/app/workers/ee/repository_import_worker.rb:9:in `perform'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/testing.rb:301:in `execute_job'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/testing.rb:296:in `block in process_job'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/testing.rb:295:in `process_job'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/testing.rb:89:in `block in raw_push'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/testing.rb:85:in `each'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/testing.rb:85:in `raw_push'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/client.rb:74:in `push'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/worker.rb:149:in `client_push'
/builds/gitlab-org/gitlab-development-kit/.bundle/ruby/2.7.0/gems/sidekiq-5.2.9/lib/sidekiq/worker.rb:93:in `perform_async'
/builds/gitlab-org/gitlab-development-kit/gitlab-development-kit/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:38:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'
/builds/gitlab-org/gitlab-development-kit/gitlab-development-kit/gitlab/app/workers/concerns/application_worker.rb:61:in `perform_async'
/builds/gitlab-org/gitlab-development-kit/gitlab-development-kit/gitlab/app/models/project.rb:1065:in `add_import_job'
/builds/gitlab-org/gitlab-development-kit/gitlab-development-kit/gitlab/ee/app/models/ee/project.rb:419:in `add_import_job'
eg.
- https://gitlab.com/gitlab-org/gitlab-development-kit/-/jobs/1317697866#L3327
- https://gitlab.com/gitlab-org/gitlab-development-kit/-/jobs/1317697863#L3327
- https://gitlab.com/gitlab-org/gitlab-development-kit/-/jobs/1317697865#L3707
In the praefect-gitaly-0
logs I see lots of:
level=fatal msg="load config: config_path \"/home/gdk/gitlab-development-kit/gitlab-development-kit/gitlab-development-kit/gitaly/gitaly-0.praefect.toml\": socket could not be created in /home/gdk/gitlab-development-kit/gitlab-development-kit/gitlab-development-kit/tmp/praefect/gitaly-0: listen unix /home/gdk/gitlab-development-kit/gitlab-development-kit/gitlab-development-kit/tmp/praefect/gitaly-0/test-27f99de0.sock: bind: invalid argument"
Followed by these messages in the praefect
log:
{"component":"HealthManager","error":"rpc error: code = Unavailable desc = connection error: desc = \"transport: Error while dialing dial unix /home/gdk/gitlab-development-kit/gitlab-development-kit/gitlab-development-kit/gitaly-praefect-0.socket: connect: no such file or directory\"","level":"error","msg":"failed checking node health","pid":6495,"storage":"praefect-internal-0","time":"2021-06-24T08:26:43.209Z","virtual_storage":"default"}
Things to try
-
❌ Try alternate Docker drivers such asaufs
anddevicemapper
-
❌ Try overlay2 Docker driver - I believe this actually is the default anyway. -
❌ General debug-
❌ Go back to previous Docker imageregistry.gitlab.com/gitlab-org/gitlab-build-images:ruby-2.7.2.patched-golang-1.16-git-2.31-lfs-2.9-chrome-89-node-14.15-yarn-1.22-postgresql-12-graphicsmagick-1.3.36
-
❌ Tune down webpack as much as possible -
❌ Try running everything as root
-
-
Try reducing the length of the path where gdk install
lands as we have seen issues in the past with gitaly and path length. -
Is webpack consuming all available file handles ?
Edited by Ash McKenzie