Support DotEnv Variables through report type artifact
What does this MR do?
This MR allows users to persist user-defined variables into ci_build_dotenv_variables
table by using dotenv
report artifact. The persisted variables are used when the system expands an environment URL. In the future, we're going to extend the usage to Inherit environment variables from dependent jobs
Related #17066 (closed)
Process Flow
Job Start, Parse artifact and Persist the result
- Runner requests a job.
POST api/v4/jobs/:id/request
. Rails presents the job as it requires dotenv file as a report-type artifact. - Runner uploads the dotenv file to object storage and notifies to Rails. (
POST api/v4/jobs/:id/artifacts
) - Rails creates a new record to
ci_job_artifacts
table viaCreateJobArtifactsService
. - Rails parses the uploaded artifact and persist the parsed data into a dedicated table.
Persisted Data Consumption
- Runner updates the
ci_builds.status
fromrunning
tosuccess
atPOST api/v4/jobs/:id
-
Ci::Build.state_machine
hooks the event to transitions adeployment.status
fromrunning
tosuccess
. -
Deployment.state_machine
hooks the event and invokesDeployments::SuccessWorker
. - In the
Deployments::SuccessWorker
, we setsenvironments.external_url
, which is expanded frombuild.options[:environment][:url]
with its variables. For example, if the original URL ishttps://$DYNAMIC_URL
and there is a key/value pair in dotenv file thatDYNAMIC_URL=awesome.com
, theenvironments.external_url
will behttps://awesome.com
.
Feature Flag
This feature is implemented behind ci_dotenv_variables
feature flag and disabled by default.
Once we've confirmed the feature is deemed stable, we remove the feature flag in order to publish the feature as GA.
TODO
-
Local QA -
app/models/ci/bridge.rb -
app/models/ci/build.rb -
app/models/ci/builds/dotenv_variable.rb -
app/models/ci/job_artifact.rb -
app/models/commit_status_enums.rb -
app/models/concerns/ci/contextable.rb -
app/presenters/commit_status_presenter.rb -
lib/gitlab/ci/config/entry/reports.rb -
lib/gitlab/ci/parsers.rb -
lib/gitlab/ci/status/build/failed.rb -
lib/gitlab/ci/parsers/dotenv_variable.rb -
app/services/ci/parse_dotenv_artifact_service.rb -
lib/api/runner.rb
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. - [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team