Improve error handling and logging in pages migration
What does this MR do?
This rake task is used to migrate files stored on disk to object storage.
It just enumerates all the projects which are not yet migrated, zips content and uploads it to the PagesDeployment
model.
Currently, all errors are reported only to Sentry: https://sentry.gitlab.net/gitlab/gitlabcom/?query=is%3Aunresolved+%22app%2Fservices%2Fpages%22, and all invalid files/links on disk are simply ignored.
This MR makes these errors:
- be reported to the migration output
- prevent project from being migrated.
Our plan is:
- delete all already migrated data using
clean_migrated_zip_storage
introduced in !53727 (merged) - rerun the migration with 'PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=false' (by default)
- inspect all the errors in logs
- rerun the migration with 'PAGES_MIGRATION_IGNORE_INVALID_ENTRIES=false' when we'll be sure that all the errors are legit
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Related to #321037 (closed)
Edited by Kamil Trzciński (Back 2025-01-01)