Rate limit Bitbucket Cloud importer

What does this MR do and why?

Adds an exponential backoff to the Bitbucket Cloud importer's API client behind a feature flag.

We had 227 errors due to OAuth2::Error, Rate limit for this resource has been exceeded.

Having these errors in a worker means that the worker is retried using Sidekiq settings which can potentially resolve the response or cause more rate limit errors due to immediate additional requests.

This MR introduces exponential backoff within the API request which retries up to 3 times and then raises an error if it's not resolved by then.

The rate limit implementation for the GitHub importer is similar but it has access to usage info from the octokit client such as octokit.rate_limit.limit and octokit.rate_limit.remaining to calculate how long to wait until retrying.

A note on the values chosen

We have the following values:

• max retries of 3
• initial delay of 1 second
• delay of delay *= 2 * (1 + Random.rand) which will increase on every retry.

Next steps

Feature flag rollout: #432379 (closed)

A follow-up MR introduces a circuit breaker to prevent additional API calls if a threshold of errors is reached. Use Gitlab circuit breaker (!137531 - merged).

Screenshots or screen recordings

Logs:

How to set up and validate locally

1. Follow https://docs.gitlab.com/ee/integration/bitbucket.html to setup OAuth for BitBucket Cloud. You will need an account on https://bitbucket.org/ that uses the same email address as the account on the gdk instance. You may need to create a new user on the gdk for this.

2. Make sure you add the bitbucket configuration in the development section of your config/gitlab.yml as shown at https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/gitlab-oauth2.md#set-up-gdk:

   development:
     <<: *base
     omniauth:
       providers:
       - { name: 'bitbucket',
           app_id: '...',
           app_secret: '...' }

3. Log into the gdk with the relevant user credentials and in ...-/profile/account connect bitbucket there:

4. Create a project and repo on BitBucket. Create an MR and add a note with a reference to issues https://bitbucket.org/<project>/<repo>/issues and specific issues.
5. Enable the feature flags: Feature.enable(:bitbucket_parallel_importer) and Feature.enable(:bitbucket_importer_exponential_backoff)
6. On your gdk instance, create a new project > click on Import project > Bitbucket Cloud > follow instructions to connect to https://bitbucket.org/.
7. Import the project.
8. In a rails console, grab the import credentials: credentials = Project.last.import_data.credentials
9. Test that the client still works by making an API request e.g. Bitbucket::Client.new(credentials).connection.get("/repositories/<your namespace>")

Simulate an error

1. Change line 29 in lib/bitbucket/connection.rb to raise OAuth2::Error.new("Rate limit for this resource has been exceeded")
2. Reload and make the API request again. The request should hang a bit and then you should see a Bitbucket::ExponentialBackoff::RateLimitError error raised due to the maximum retries being exceeded. You will also see two logs in tail -f log/importer.log showing Retrying in x seconds.

[Optional] Cause a rate limit error

1. Run multiple threads of API requests in parallel to cause a rate limit, e.g.

   100.times.with_index { |index| puts index; Bitbucket::Client.new(credentials).connection.get("/repositories/<your namespace>") }

2. View the logs for the retries.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.

Related to #251222 (closed)