Update Project/Group Exporter temp folder cleanup
What does this MR do?
Project/Group Export (https://docs.gitlab.com/ee/user/project/settings/import_export.html) temporarily exports data into shared temp folder before uploading it to object storage/system uploads with Carrierwave.
It has been reported in #297579 (closed) that project export repeatedly fails when initiated by multiple users at the same time (using project creation using custom template) with #<Errno::ENOENT: No such file or directory @ rb_sysopen error.
It's happening due to a change I made 7 months ago !32326 (merged) as part of #194193 (closed) which started to cleanup a 'base' temp directory of a project/group, which can remove any concurrent export job's temporary files as well as a side effect. This causes bad experience for users exporting a project at the same time, reducing export success chance. In the case of mentioned issue, 200 users create a project from custom template, which utilizes import/export functionality under the hood.
This MR fixes this issue and cleans up 'archive path' for a project/group, instead of a 'base path'. This way temp files for individual export are removed, without affecting any other concurrent exports.
A typical project path during export looks like this:
# Base path, `project.disk_path` for project, `group.full_path` for group
gitlab/shared/tmp/gitlab_exports/@hashed/c75de23d89df36ba921287616ee8edb4c986e328a78e033e57c1e5e2b59c838e
# Archive path, where gzipped archive is placed once export is complete
gitlab/shared/tmp/gitlab_exports/@hashed/c75de23d89df36ba921287616ee8edb4c986e328a78e033e57c1e5e2b59c838e/<SecureRandom.hex>
# Export path, where exported files are put while export is in progress
gitlab/shared/tmp/gitlab_exports/@hashed/c75de23d89df36ba921287616ee8edb4c986e328a78e033e57c1e5e2b59c838e/<SecureRandom.hex>/<SecureRandom.hex>For import, cleaning up 'base path' is still desirable, as there is no concurrency involved.
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec -
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team