Add issues importer
What does this MR do?
Adds implementation of the issues importer workers and services.
Import flow
Import is based on several import stages: StartImportWorker, ImportLabelsWorker, ImportIssuesWorker, ImportAttachmentsWorker, ImportNotesWorker and FinishImportWorker
This MR touches only ImportIssuesWorker
and partly FinishImportWorker
stages. Other stages are empty and just pass the work to next stage without doing any actual work.
Flow Steps
- Import is triggered by setting project.import_state to
scheduled
state, with aforce-import
value set inproject.import_data
. We need theforce-import
flag to differentiate between jira import vs mirror sync, that uses sameproject.import_state
object. - When
project.import_state
it will schedule aStartImportWorker
that will actually start the import -
StartImportWorker
schedules theImportLabelsWorker
-
ImportLabelsWorker
after importing labels(its empty at this point) schedulesImportIssuesWorker
-
ImportIssuesWorker
calls jira api and pulls a batch of issues, batch size is set to 1000. For each issue aImportIssuesWorker
is scheduled that actually saves data to DB. - If there are more than 1000 issues to be imported the worker will not schedule next stage, but rather schedule the
ImportIssuesWorker
again after first batch of 1000ImportIssuesWorker
workers finish. - When
ImportIssuesWorker
reaches last page it will pass on to next stage and schedule theImportAttachmentsWorker
-
ImportAttachmentsWorker
->ImportNotesWorker
-
ImportNotesWorker
->FinishImportWorker
-
FinishImportWorker
sets theproject.import_state
tofinished
and does some cleanup as for instance removingforce-import
flag so that if mirror sync triggers a sync it will not perceive it as an import.
Important
- All workers are behind
jira_issue_import
feature flag, which is disabled by default.
Screenshots
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry -
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by Alexandru Croitor