Skip to content

Draft: Fix and clarify `filter`-parameter usage on group-level CI variable API

Manuel Grabowski requested to merge mg-group-variables-scope-filter-20231109 into master

Draft because it is a breaking change and needs a feature flag

What does this MR do and why?

It's potentially confusing, so I'll clarify first: This is about updating group-level CI variables based on their scope (and key), which is currently not possible. It is not about updating their scope, which works perfectly fine.

This fixes Add environment_scope filter to group ci-variab... (#340185 - closed) – however, while working on this I realized that the issue isn't 100% correct. The filter parameter is supported in the group-level CI variables API. As far as I can tell, this has always the case.

In Add environment scope to group CI variables API... (!55573 - merged) we orginially added environment_scope support to the group-level CI variables API. Looking at the tests in that MR, filter[environment_scope] was already being used then. Furthermore, both GET and DELETE on that endpoint do process (and actually expect!) filter[environment_scope] when testing on GitLab.com today.

However, there is two caveats:

  • The group-level CI variable docs have never been updated to include details about filter[environment_scope] usage (compare to the project-level docs)
  • The PUT method on that endpoint (to update a group-level CI variable) does not support filter[environment_scope]
    • I think this was an oversight, and because there is no tests for that specific case it wasn't noticed

This MR adds filter[environment_scope] support for PUT, documents filter[environment_scope] for all three supported methods on the API endpoint, and adds some tests to cover the case where a variable with a specific key / environment_scope combination is updated.

There's two changes in behavior with this MR:

  • When trying to update the environment_scope of a variable to a value that is already taken by another variable, the API now returns :conflict instead of :bad_request.
  • When trying to update a variable that exists multiple times with different scopes and not providing filter[environment_scope], you will now get a :conflict response (and a message telling you to use filter[environment_scope]). Before this change, we would simply have updated the one of the variables – there was no way to target a specific scope. As this old behavior was already broken, fixing it doesn't constitute a breaking change.

Screenshots or screen recordings

See below, I'll include sample output responses in the reproduction steps.

How to set up and validate locally

As a baseline, here's how GitLab.com behaves today:

# Create "test" variable with "scope1"
$> curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables" --form "key=test" --form "value=val1" --form "environment_scope=*/scope1"
{"variable_type":"env_var","key":"test","value":"val1","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope1","description":null}

# Request "test" variable, get response
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test"
{"variable_type":"env_var","key":"test","value":"val1","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope1","description":null}

# Create "test" variable with "scope2"
$> curl --request POST --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables" --form "key=test" --form "value=val2" --form "environment_scope=*/scope2"
{"variable_type":"env_var","key":"test","value":"val2","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope2","description":null}

# Request "test" variable again, get error response
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test"
{"message":"There are multiple variables with provided parameters. Please use 'filter[environment_scope]'"}

# Request scoped "test" variables, get correct value/responses
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test" --form "filter[environment_scope]=*/scope1"
{"variable_type":"env_var","key":"test","value":"val1","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope1","description":null}
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test" --form "filter[environment_scope]=*/scope2"
{"variable_type":"env_var","key":"test","value":"val2","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope2","description":null}

# !! Update "test" variable without specifying scope, observe value of variable with "scope1" being updated
$> curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test" --form "value=NEWVAL"
{"variable_type":"env_var","key":"test","value":"NEWVAL","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope1","description":null}

# Verify current values for both scopes
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test" --form "filter[environment_scope]=*/scope1"
{"variable_type":"env_var","key":"test","value":"NEWVAL","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope1","description":null}
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test" --form "filter[environment_scope]=*/scope2"
{"variable_type":"env_var","key":"test","value":"val2","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope2","description":null}

# Delete "test" variable, get error message
$> curl --request DELETE --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test"
{"message":"There are multiple variables with provided parameters. Please use 'filter[environment_scope]'"}

# Delete "test" variable with scope
$> curl --request DELETE --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "https://gitlab.com/api/v4/groups/71715194/variables/test" --form "filter[environment_scope]=*/scope1"
(no response message, but it is successfully deleted)

As we can see: Both GET and DELETE not only accept filter[environment_scope] but in fact require it when the request would otherwise be ambiguous – this is currently not documented. However, for PUT (marked with !! above) the ambiguity is ignored and the oldest variable with that key is updated.

In this branch, the behavior for PUT changes as follows, everything else is identical:

# Update "test" variable without specifying scope, get error
$> curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "http://gdk.test:3000/api/v4/groups/74/variables/test" --form "value=NEWVAL"
{"message":"There are multiple variables with provided parameters. Please use 'filter[environment_scope]'"}

# Update "test" variable with "scope2", observe value of variable with "scope2" being updated
$> curl --request PUT --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "http://gdk.test:3000/api/v4/groups/74/variables/test" --form "value=NEWVAL" --form "filter[environment_scope]=*/scope2"
{"variable_type":"env_var","key":"test","value":"NEWVAL","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope2","description":null}

# Verify current values for both scopes
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "http://gdk.test:3000/api/v4/groups/74/variables/test" --form "filter[environment_scope]=*/scope1"
{"variable_type":"env_var","key":"test","value":"val1","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope1","description":null}
$> curl --request GET --header "PRIVATE-TOKEN: $GITLAB_TOKEN" "http://gdk.test:3000/api/v4/groups/74/variables/test" --form "filter[environment_scope]=*/scope2"
{"variable_type":"env_var","key":"test","value":"NEWVAL","protected":false,"masked":false,"raw":false,"environment_scope":"*/scope2","description":null}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Manuel Grabowski

Merge request reports