Optimize variables on pipeline creation for yaml and node
What does this MR do and why?
On pipeline creation, a stub build is created for every job candidate to run rules against it. This is an expensive operation for many reasons. More details are at #350057 (closed).
One of the downsides of using a stub build is that variable calculation becomes also expensive because it tries to reach sub-relations, for example "metadata" in the scope of this change.
Eventually, our goal is to not use a stub build at all, but I'd like to gradually lose the connection between the AR object and variables. And, this is the first step; we are generating yaml_variables and node_variables from the hash instead of the AR object.
More information about the performance optimization: &14348 (comment 1982576223)
This change is behind the feature flag
ci_variables_optimization_for_yaml_and_node (#470827 (closed)).
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.