Skip to content

Submit exit code back to Rails when a job fails

Marius Bobin requested to merge submit-exit-code-back-to-rails into master

What does this MR do?

In gitlab#273157 (closed) we want to allow users to control allow_failure functionality using the job's exit code:

test_job:
  script:
    - execute_script_that_will_fail
  allow_failure:
    exit_codes:
      - 137
      - 255

To do this, we need to capture and send the exit code back to Rails.

Why was this MR needed?

What's the best way to test this MR?

Kubernetes executors

  1. Have the following .gitlab-ci.yml

    .gitlab-ci.yml 
    image: busybox:latest

test1:
  variables:
    FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: 1
  script:
    - exit 42

test2:
  variables:
    FF_USE_LEGACY_KUBERNETES_EXECUTION_STRATEGY: 0
  script:
    - exit 42

  2. Configure a k8s runner

    K8s runner configuration example 
    [[runners]]
  name = "k8s"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxx"
  executor = "kubernetes"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.kubernetes]
    host = ""
    bearer_token_overwrite_allowed = false
    image = ""
    helper_image = "gitlab/gitlab-runner-helper:x86_64-af099ad9"
    namespace = ""
    namespace_overwrite_allowed = ""
    privileged = false
    service_account_overwrite_allowed = ""
    pod_annotations_overwrite_allowed = ""
    [runners.kubernetes.affinity]
    [runners.kubernetes.pod_security_context]
    [runners.kubernetes.volumes]

  3. Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code

  4. Build executable for your arch and OS: make runner-and-helper-bin-host

  5. Start the runner: out/binaries/gitlab-runner --debug run

  6. The exit code should appear in the logs two times:

    Example log entry 
    {"time":"2020-12-02T10:19:19.072Z","severity":"INFO","duration_s":0.0482,"db_duration_s":0.01883,"view_duration_s":0.02937,"status":200,"method":"PUT","path":"/api/v4/jobs/5818","params":[{"key":"info","value":{"name":"gitlab-runner","version":"13.7.0~beta.92.g0f7d48f1","revision":"0f7d48f1","platform":"darwin","architecture":"amd64","executor":"kubernetes","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null}}},{"key":"token","value":"[FILTERED]"},{"key":"state","value":"failed"},{"key":"failure_reason","value":"script_failure"},{"key":"checksum","value":"crc32:ebc1bb65"},{"key":"output","value":"[FILTERED]"},{"key":"exit_code","value":42}],"host":"gdev.bobin.ro","remote_ip":"192.168.22.142","ua":"gitlab-runner 13.7.0~beta.92.g0f7d48f1 (submit-exit-code-back-to-rails; go1.15.5; darwin/amd64)","route":"/api/:version/jobs/:id","queue_duration_s":0.281487,"redis_calls":5,"redis_duration_s":0.001596,"redis_read_bytes":379,"redis_write_bytes":3371,"redis_cache_calls":2,"redis_cache_duration_s":0.001093,"redis_cache_read_bytes":205,"redis_cache_write_bytes":168,"redis_queues_calls":3,"redis_queues_duration_s":0.000503,"redis_queues_read_bytes":174,"redis_queues_write_bytes":3203,"correlation_id":"01ERHFD9TNE82WJ7EPSACKS4TV","meta.user":"root","meta.project":"root/k8s-exec","meta.root_namespace":"root","meta.caller_id":"/api/:version/jobs/:id","meta.feature_category":"continuous_integration","content_length":"763"}

Note: when using attach strategy to execute scripts the job is failing with a timeout error: ERROR: Job failed (system failure): prepare environment: waiting for pod running: timed out waiting for pod to start. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information I think this is related to my config.

Docker executor

  1. Have the following .gitlab-ci.yml

    .gitlab-ci.yml 
    image: busybox:latest

test1:
  script:
    - exit 42

  2. Configure a Docker runner

    Docker runner configuration example 
    [[runners]]
  name = "gdev.bobin.ro"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "alpine"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

  3. Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code

  4. Build executable for your arch and OS: make runner-and-helper-bin-host

  5. Start the runner: out/binaries/gitlab-runner --debug run

  6. The exit code should appear in the logs:

    Example log entry

Shell executor

  1. Have the following .gitlab-ci.yml

    .gitlab-ci.yml 
    test1:
  script:
    - exit 42

  2. Configure a shell runner

    Shell runner configuration example 
    [[runners]]
  name = "shell"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxx"
  executor = "shell"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

  3. Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code

  4. Build executable for your arch and OS: make runner-and-helper-bin-host

  5. Start the runner: out/binaries/gitlab-runner --debug run

  6. The exit code should appear in the logs:

    Example log entry 
    {"time":"2020-12-02T11:26:18.747Z","severity":"INFO","duration_s":0.0899,"db_duration_s":0.05391,"view_duration_s":0.03599,"status":200,"method":"PUT","path":"/api/v4/jobs/5824","params":[{"key":"info","value":{"name":"gitlab-runner","version":"13.7.0~beta.92.g0f7d48f1","revision":"0f7d48f1","platform":"darwin","architecture":"amd64","executor":"shell","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null}}},{"key":"token","value":"[FILTERED]"},{"key":"state","value":"failed"},{"key":"failure_reason","value":"script_failure"},{"key":"checksum","value":"crc32:13e5809a"},{"key":"output","value":"[FILTERED]"},{"key":"exit_code","value":42}],"host":"gdev.bobin.ro","remote_ip":"192.168.22.142","ua":"gitlab-runner 13.7.0~beta.92.g0f7d48f1 (submit-exit-code-back-to-rails; go1.15.5; darwin/amd64)","route":"/api/:version/jobs/:id","queue_duration_s":0.23975,"redis_calls":4,"redis_duration_s":0.000912,"redis_read_bytes":176,"redis_write_bytes":3314,"redis_cache_calls":1,"redis_cache_duration_s":0.00027,"redis_cache_read_bytes":2,"redis_cache_write_bytes":112,"redis_queues_calls":3,"redis_queues_duration_s":0.000642,"redis_queues_read_bytes":174,"redis_queues_write_bytes":3202,"correlation_id":"01ERHK7Z9HJ3Q42B3JNNTW9MXY","meta.user":"root","meta.project":"root/k8s-exec","meta.root_namespace":"root","meta.caller_id":"/api/:version/jobs/:id","meta.feature_category":"continuous_integration","content_length":"760"}

SSH executor

  1. Have the following .gitlab-ci.yml

    .gitlab-ci.yml 
    test1:
  script:
    - exit 42

  2. Configure a shell runner

    Shell runner configuration example 
    [[runners]]
  name = "ssh"
  url = "http://gdev.bobin.ro:3000/"
  token = "xxxxxx"
  executor = "ssh"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
    [runners.cache.azure]
  [runners.ssh]
    user = "root"
    host = "gdev.bobin.ro"
    port = "2222"
    identity_file = "/Users/marius/.ssh/id_ed25519"

  3. Start a Docker container with SSH service enabled: docker run -it --rm -p 2222:22 -e SSH_ENABLE_ROOT=true -v ${HOME}/.ssh/id_ed25519.pub:/root/.ssh/authorized_keys:ro docker.io/panubo/sshd:1.3.0

  4. Tail the GitLab API log file using tail -f log/api_json.log | grep exit_code

  5. Build executable for your arch and OS: make runner-and-helper-bin-host

  6. Start the runner: out/binaries/gitlab-runner --debug run

  7. The exit code should appear in the logs:

    Example log entry 
    {"time":"2020-12-02T11:35:30.150Z","severity":"INFO","duration_s":0.04323,"db_duration_s":0.01711,"view_duration_s":0.02612,"status":200,"method":"PUT","path":"/api/v4/jobs/5825","params":[{"key":"info","value":{"name":"gitlab-runner","version":"13.7.0~beta.92.g0f7d48f1","revision":"0f7d48f1","platform":"darwin","architecture":"amd64","executor":"ssh","shell":"bash","features":{"variables":"[FILTERED]","image":null,"services":null,"artifacts":null,"cache":null,"shared":null,"upload_multiple_artifacts":null,"upload_raw_artifacts":null,"session":null,"terminal":null,"refspecs":null,"masking":null,"proxy":null,"raw_variables":"[FILTERED]","artifacts_exclude":null,"multi_build_steps":null,"trace_reset":"[FILTERED]","trace_checksum":"[FILTERED]","trace_size":"[FILTERED]","vault_secrets":"[FILTERED]","cancelable":null}}},{"key":"token","value":"[FILTERED]"},{"key":"state","value":"failed"},{"key":"failure_reason","value":"script_failure"},{"key":"checksum","value":"crc32:4ababebe"},{"key":"output","value":"[FILTERED]"},{"key":"exit_code","value":42}],"host":"gdev.bobin.ro","remote_ip":"192.168.22.142","ua":"gitlab-runner 13.7.0~beta.92.g0f7d48f1 (submit-exit-code-back-to-rails; go1.15.5; darwin/amd64)","route":"/api/:version/jobs/:id","queue_duration_s":0.236205,"redis_calls":4,"redis_duration_s":0.000761,"redis_read_bytes":176,"redis_write_bytes":3314,"redis_cache_calls":1,"redis_cache_duration_s":0.00028,"redis_cache_read_bytes":2,"redis_cache_write_bytes":112,"redis_queues_calls":3,"redis_queues_duration_s":0.000481,"redis_queues_read_bytes":174,"redis_queues_write_bytes":3202,"correlation_id":"01ERHKRSTE7DMX1JF027FKCV81","meta.user":"root","meta.project":"root/k8s-exec","meta.root_namespace":"root","meta.caller_id":"/api/:version/jobs/:id","meta.feature_category":"continuous_integration","content_length":"760"}

What are the relevant issue numbers?

gitlab#273157 (closed)

Edited by Marius Bobin

Merge request reports