Adjust Bitbucket Cloud issues worker to be resumable
-
Please check this box if this contribution uses AI-generated content (including content generated by GitLab Duo features) as outlined in the GitLab DCO & CLA
What does this MR do and why?
Adjust Bitbucket Cloud issues worker to be resumable. This is a continuation from this MR, but for "Issue Worker".
This MR also change the feature flag from wip
to beta
. Quoting from the doc.
Once the feature is complete, the feature flag type can be changed to the gitlab_com_derisk or beta type depending on how the feature will be presented/documented to customers.
There are 2 more components to be worked on:
pull_requests_notes
issues_notes
But both of them need a refactoring to move the API call from "Importer Worker" to "Stage Worker", similar with this issue.
Changelog: performance
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
Successful imported issues: | |
Interruption happened after issue title issue 40 : |
|
Upon resume, retrying issue title issue 40 . Then continuing to MR title issue 41 : |
|
How to set up and validate locally
-
Setup Bitbucket Cloud following this guide
-
Enable issue tracker by following this guide
-
In rails console enable the feature flag
Feature.enable(:bitbucket_import_resumable_worker)
-
Patch the code to add interruption:
diff --git a/lib/gitlab/bitbucket_import/parallel_scheduling.rb b/lib/gitlab/bitbucket_import/parallel_scheduling.rb
index c4703c71da9a..7a89b2232a80 100644
--- a/lib/gitlab/bitbucket_import/parallel_scheduling.rb
+++ b/lib/gitlab/bitbucket_import/parallel_scheduling.rb
@@ -42,6 +42,16 @@ def each_object_to_import
options = collection_options.merge(next_url: page_keyset.current)
client.each_page(collection_method, representation_type, repo, options) do |page|
+ if collection_method == :issues
+ log_info(message: page.inspect)
+ Gitlab::Redis::SharedState.with do |redis|
+ temp_key = 'test-bitbucket-pr'
+ temp_counter = redis.incr(temp_key)
+ redis.expire(temp_key, 5.minutes)
+ raise "purposely interrupt" if temp_counter == 2
+ end
+ end
+
page.items.each do |object|
job_waiter.jobs_remaining = Gitlab::Cache::Import::Caching.increment(job_waiter_remaining_cache_key)
- Visit Bitbucket Cloud import page http://127.0.0.1:3000/import/bitbucket/status
- Click the
Import
button. - Tail the log file:
tail -f log/importer.log
Related to #448895