Backend: Stop expanding file variables when sending to Runner

Summary

This issue is extracted from #29407 (closed)

From the comments #29407 (comment 947110258) and #29407 (comment 989127953)

Step	Status
1. GitLab: Stop expanding file variables when sending to Runner	👈 You are here
2. Runner: Refactor the creation of the job temporary file path	gitlab-runner#29128 (closed)
3. Fix file variables in Runner	#29407 (closed)

TODOs:

• GitLab needs to start sending file variables to Runner non-expanded because Runner already does this.
• Use a feature flag.

Technical proposal

from: #29407 (comment 906397647)

Added some file variables;

Used this config YAML which includes some examples from comments in this and another issue;

variables:
    EXTRA_ARGS: "-f $TEST_FILE"
    DOCKER_REMOTE_ARGS: --tlscacert="$DOCKER_CA_CERT"
    EXTRACTED_CRT_FILE: ${DOCKER_CA_CERT}.crt
    MY_FILE_VAR: $TEST_FILE

test:
    script:
        - echo "run something $EXTRA_ARGS"
        - echo "docker run $DOCKER_REMOTE_ARGS"
        - echo "run --output=$EXTRACTED_CRT_FILE"
        - echo "Will read private key from $MY_FILE_VAR"

Currently

When Runner asks the job, GitLab send the job variables in this format;

 {:key=>"TEST_FILE", :value=>"hello, this is test", :public=>false, :file=>true, :masked=>false},
 {:key=>"EXTRA_ARGS", :value=>"-f hello, this is test", :public=>true, :masked=>false},
 {:key=>"DOCKER_CA_CERT", :value=>"BEGIN\nthis is secret\nEND", :public=>false, :file=>true, :masked=>false},
 {:key=>"DOCKER_REMOTE_ARGS", :value=>"--tlscacert=\"BEGIN\nthis is secret\nEND\"", :public=>true, :masked=>false},
 {:key=>"EXTRACTED_CRT_FILE", :value=>"BEGIN\nthis is secret\nEND.crt", :public=>true, :masked=>false},
 {:key=>"MY_FILE_VAR", :value=>"hello, this is test", :public=>true, :masked=>false},

As you can see, file variables are expanded into YAML variables.

Job result:

Proposal

With this POC change;

--- a/lib/gitlab/ci/variables/collection.rb
+++ b/lib/gitlab/ci/variables/collection.rb
@@ -78,7 +78,7 @@ def expand_value(value, keep_undefined: false)
             if match[:key]
               # we matched variable
               if variable = self[match[:key]]
-                variable.value
+                variable.file? ? match[0] : variable.value
               elsif keep_undefined
                 match[0]
               end

GitLab start sending variables of the job like this;

 {:key=>"TEST_FILE", :value=>"hello, this is test", :public=>false, :file=>true, :masked=>false},
 {:key=>"EXTRA_ARGS", :value=>"-f $TEST_FILE", :public=>true, :masked=>false},
 {:key=>"DOCKER_CA_CERT", :value=>"BEGIN\nthis is secret\nEND", :public=>false, :file=>true, :masked=>false},
 {:key=>"DOCKER_REMOTE_ARGS", :value=>"--tlscacert=\"$DOCKER_CA_CERT\"", :public=>true, :masked=>false},
 {:key=>"EXTRACTED_CRT_FILE", :value=>"${DOCKER_CA_CERT}.crt", :public=>true, :masked=>false},
 {:key=>"MY_FILE_VAR", :value=>"$TEST_FILE", :public=>true, :masked=>false},

As you can see, file variables are not expanded. However, now they are expanded by Runner;