Skip to content

Introduce a "Pipeline specific API" that could be used in CI/CD jobs.

During discussion with @bikebilly we've got an idea of introducing a pipeline specific API, that could be used inside of CI/CD jobs. Currently we can do a lot of things with API, but it's available in the global scope.

To describe what pipeline specific means, let's consider an example:

In job deploy application we execute commands that are deploying our application on some server. We also configure there the environment settings, where the URL is most interesting for us:

deploy application:
  stage: deploy
  environment:
    name: development-$CI_COMMIT_REF_SLUG
    url: $CI_COMMIT_REF_SLUG.my-application.com
  script:
  - cloud-tool deploy app.json --url $CI_ENVIRONMENT_URL

Now, in next stage we want to have a post-deploy tests job, that uses the deployed review app and performs some checks on it:

post-deploy tests:
  stage: post-deploy
  scripts:
  - curl POST $URL -F test_field=test_value

The obvious problem here is how to get the $URL value? We have few possibilities, but none is a clean one:

  1. We can build the variable manually in another job, e.g. with export URL=$CI_COMMIT_REF_SLUG.my-application.com. This is good, but what if we want to use for example $CI_ENVIRONMENT_SLUG as part of the URL? It's not available in post-deploy tests and to create it manually we would need to reproduce GitLab's internal slugify methods inside of post-deploy tests job.

  2. We can store the value of $CI_ENVIRONMENT_URL as artifact in deploy application job, and then get it from the artifact file in post-deploy tests job. But this seems to be a nasty workaround. Using artifacts for something such simple as get the url that's already defined in .gitlab-ci.yml seems to be an overuse of such feature.

  3. We can use the API, but for this we need to

    • pass some access token with secret variables
    • iterate over all environments returned from environments API, to find out the one we're looking for.

The pipeline specific API that I'm proposing would simplify this, by having all pipeline related data organized at one API endpoint. The above problem we could resolve with something like:

post-deploy tests:
  stage: post-deploy
  scripts:
  - export URL=$(curl -X GET -H "Private-Token: $CI_JOB_TOKEN" $CI_SERVER/api/v4/pipeline/$CI_PIPELINE_ID/environments/deploy%20application/url)
  - curl POST $URL -F test_field=test_value

So we're accessing environment's URL calling GitLab's API at /api/v4/pipeline/:pipeline_id/environments/:job_name/url and using $CI_JOB_TOKEN as authorization token. If we would like to have the slug of the environment's name, we could ask for /api/v4/pipeline/:pipeline_id/environments/:job_name/slug. And for all data about specified environment: /api/v4/pipeline/:pipeline_id/environments/:job_name.

Such pipeline specific API opens also other possibilities. We were commonly asked if it's possible to use variables/data defined during jobs as part of environment setting. And currently the only answer is NO, because environment is configured only on GitLab's side. But with such API we could have such possibility.

Let's get back to the above example. We've used an abstract cloud-tool deploy command to deploy some application to some cloud, setting some specific URL. Let's now consider a situation, where such tool doesn't allow us to specify the URL, but it returns the URL as a result (and the URL itself is not predictable). We could then define our job as:

deploy application:
  stage: deploy
  environment:
    name: development-$CI_COMMIT_REF_SLUG
  script:
  - cloud-tool deploy app.json --url $CI_ENVIRONMENT_URL | tee output.txt
  - url=$(cat output.txt | grep -E "url=[^ ]+" | sed "s|url=||")
  - curl -X PUT -H "Private-Token: $CI_JOB_TOKEN" $CI_SERVER/api/v4/pipeline/$CI_PIPELINE_ID/environments/deploy%20application -F "url=$url"

With this job we're deploying the application and writing the output in output.txt file, next we're extracting the created URL from the output and finally we're setting this URL as the value of CI_ENVIRONMENT_URL of defined environment.