Remove docker-machine feature flag
What does this MR do?
Remove FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE
feature flag
Why was this MR needed?
Remove FF_SKIP_DOCKER_MACHINE_PROVISION_ON_CREATION_FAILURE
since it's
been enabled by default for a month and no issues were reported. The
feature flag was added as a fail safe in case there are problems in
GitLab.com. Since we saw positive
impact
we should never try and provision a machine.
What's the best way to test this MR?
There is no easy way to force a creation failure mid creation, and it's not always reproducible because it needs to be at the moment where the local state exists and remote state doesn't. Below is an attempt at this.
-
Start
docker+machine
runner configured like below to provision VMs on GCP.config.toml
[[runners]] name = "docker-machine" url = "https://gitlab.com/" token = "" executor = "docker+machine" [runners.docker] tls_verify = false image = "alpine:3.14" privileged = false disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/cache"] shm_size = 0 [runners.machine] IdleCount = 0 MachineDriver = "google" MachineName = "steveazz-%s" MachineOptions = [ "google-project=group-verify-df9383", "google-zone=europe-west4-b", ]
-
Open up the GCP console with the configured project opened.
-
Start a job for this executor and as soon as you see
Waiting for Instance
in the runner logs, delete the machine in GCP -
Look at the runner system logs, and check that it failed to create a machine and created a new one
runner logs
{"driver":"google","level":"info","msg":"Creating machine...","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:48+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Generating SSH Key","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:48+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Creating host...","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:49+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Opening firewall ports","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:49+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Creating instance","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:49+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Waiting for Instance","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:35:50+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Uploading SSH Key","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:37:51+02:00"} {"driver":"google","level":"error","msg":"Error creating machine: Error in driver during machine creation: operation error: {RESOURCE_NOT_FOUND The resource 'projects/group-verify-df9383/zones/europe-west4-b/instances/runner-rducy8nc-steveazz-1629185747-3a0f80bb' was not found [] []}","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"create","time":"2021-08-17T09:38:20+02:00"} {"error":"exit status 1","fields.time":152745532205,"level":"error","msg":"Machine creation failed","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","time":"2021-08-17T09:38:20+02:00"} {"level":"warning","lifetime":152745696539,"msg":"Requesting machine removal","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","now":"2021-08-17T09:38:20.350354+02:00","reason":"Failed to create","time":"2021-08-17T09:38:20+02:00","used":152745696864,"usedCount":0} {"level":"warning","lifetime":152841008533,"msg":"Stopping machine","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","reason":"Failed to create","time":"2021-08-17T09:38:20+02:00","used":95252257,"usedCount":0} {"level":"info","msg":"Stopping \"runner-rducy8nc-steveazz-1629185747-3a0f80bb\"...","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"stop","time":"2021-08-17T09:38:20+02:00"} {"driver":"google","level":"info","msg":"Running pre-create checks...","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:21+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Check that the project exists","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:21+02:00"} {"level":"error","msg":"googleapi: Error 404: The resource 'projects/group-verify-df9383/zones/europe-west4-b/instances/runner-rducy8nc-steveazz-1629185747-3a0f80bb' was not found, notFound","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"stop","time":"2021-08-17T09:38:22+02:00"} {"error":"exit status 1","level":"warning","lifetime":154651705012,"msg":"Error while stopping machine","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","reason":"Failed to create","time":"2021-08-17T09:38:22+02:00","used":1905948841,"usedCount":0} {"level":"warning","lifetime":154651784143,"msg":"Removing machine","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","reason":"Failed to create","time":"2021-08-17T09:38:22+02:00","used":1906027514,"usedCount":0} {"level":"info","msg":"About to remove runner-rducy8nc-steveazz-1629185747-3a0f80bb","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:22+02:00"} {"level":"info","msg":"WARNING: This action will delete both local reference and remote instance.","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:22+02:00"} {"level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Deleting instance.","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:22+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Check if the instance already exists","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"} {"driver":"google","level":"info","msg":"Creating machine...","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Generating SSH Key","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Creating host...","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Opening firewall ports","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:22+02:00"} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Creating instance","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:23+02:00"} {"level":"info","msg":"(runner-rducy8nc-steveazz-1629185747-3a0f80bb) Remote instance does not exist, proceeding with removing local reference","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:23+02:00"} {"level":"info","msg":"Successfully removed runner-rducy8nc-steveazz-1629185747-3a0f80bb","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","operation":"remove","time":"2021-08-17T09:38:23+02:00"} {"level":"info","lifetime":156045999827,"msg":"Machine removed","name":"runner-rducy8nc-steveazz-1629185747-3a0f80bb","now":"2021-08-17T09:38:23.650656+02:00","reason":"Failed to create","retries":0,"time":"2021-08-17T09:38:23+02:00","used":3300243540,"usedCount":0} {"driver":"google","level":"info","msg":"(runner-rducy8nc-steveazz-1629185901-db2c619c) Waiting for Instance","name":"runner-rducy8nc-steveazz-1629185901-db2c619c","operation":"create","time":"2021-08-17T09:38:24+02:00"}
What are the relevant issue numbers?
closes #27749 (closed)