Remove expired pipeline artifacts
Ref: #234042 (closed)
Final step to clean up expired pipeline artifacts.
This feature is behind a feature flag (coverage_report_view
).
What does this MR do?
This MR creates a new service
which destroys expired pipeline_artifacts
.
It includes the following items:
-
Adds a new index for expire_at
toci_pipeline_artifacts
-
Modifies DestroyExpiredJobArtifactsService
to clean upCi::PipelineArtifact
Why are we doing this?
You can visualize how much data is being stored on with your gitlab instance by visiting this page.
Following up !39114 (merged), where we introduced a new column (expire_at
defaulted to 1 week).
As pointed by @dosuken123 when introducing our new model in !37969 (merged).
Our system was missing an expiration mechanism:
We've introduced Pipeline Artifacts but have not implemented an expiration mechanism for them. We should implement a mechanism for expiration in order to not put pressure on our storage.
This MR implements the necessary logic to destroy pipeline_artifacts
older than 1 week to reduce user' storage capacity.
Migration output
$ bin/rails db:migrate
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: migrating ==============
-- transaction_open?()
-> 0.0000s
-- index_exists?(:ci_pipeline_artifacts, :expire_at, {:name=>"index_ci_pipeline_artifacts_on_expire_at", :algorithm=>:concurrently})
-> 0.0032s
-- add_index(:ci_pipeline_artifacts, :expire_at, {:name=>"index_ci_pipeline_artifacts_on_expire_at", :algorithm=>:concurrently})
-> 0.0033s
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: migrated (0.0069s) =====
Migration Rollback
$ bin/rails db:rollback
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: reverting ==============
-- transaction_open?()
-> 0.0000s
-- indexes(:ci_pipeline_artifacts)
-> 0.0037s
-- remove_index(:ci_pipeline_artifacts, {:algorithm=>:concurrently, :name=>"index_ci_pipeline_artifacts_on_expire_at"})
-> 0.0020s
== 20200827150057 AddIndexExpireAtToPipelineArtifacts: reverted (0.0061s) =====
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry - Changelog to be included with feature flag removal.
- [-] Documentation (if required) - Documentation to be included with feature flag removal.
-
Code review guidelines - [-] Merge request performance guidelines
-
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
- [-] Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process.
- [-] Tested in all supported browsers
- [-] Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
- [-] Label as security and @ mention
@gitlab-com/gl-security/appsec
- [-] The MR includes necessary changes to maintain consistency between UI API, email, or other methods
- [-] Security reports checked/validated by a reviewer from the AppSec team