Queue workers only for objects with attachments
What does this MR do and why?
Queue GitHub attachment importer workers only for objects with attachments
During a GitHub import, when we queue a worker we spread them to run 1,000 per minute. The more workers queued, the longer the spread across time.
Because most workers will do nothing, this spread adds unnecessary time to an import. The larger the import, the more time is wasted in this stage.
This change will queue workers only for objects that have attachments.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
- In rails console enable the feature:
Feature.enable(:github_importer_attachments)
- Tail your Sidekiq logs for GitHub import attachment workers:
tail -f sidekiq_client.log | grep "GithubImport::Attachments::"
. - Ensure GitHub is enabled as an import source for your instance (docs).
- Create a GitHub project:
- Add 1 issue with just a text description
- Add 1 more issue with an image attachment in its description. Add comments to this issue:
- Add 1 comment with an attachment in its description
- Add 1 comment which is just text
- Import the GitHub project (docs). Ensure that the Import Markdown attachments (links) UI option is checked.
- Expand the
Details
section to watch the progress of the import. You should only see counts ofIssue links
,Note links
, orPR attachments
that match the counts of actual attachments. Onmaster
these counts will match the number of issues, notes and PRs regardless of whether they have attachments of note. - Ensure that the project's attachments are imported:
- Go to the imported project and right-click on image attachments to verify they are hosted locally and not on github.com.
Related to #436400 (closed)
Edited by Luke Duncalfe