Skip to content

Filter GitHub projects to import using GitHub Search API

What does this MR do?

This MR replaces GitHub Importer's filtering functionality from filtering on GitLab server side to use GitHub Search API in order to implement pagination in the future MR (as well as being able to filter results and have them paginated).

There are a few caveats:

  1. In order to compose a correct search query that includes user's repositories as well as repos from organisations and from other users (where user is a collaborator) we need to do a few network requests to fetch usernames and org names.
  2. A list of repos where user is collaborator is fetched in order to retrieve a list of usernames owning those repos. I could not find a better way of fetching this list.
  3. A list of organisations is fetched to add to the overall search query
  4. GitHub search API rate limit is 30 per minute https://developer.github.com/v3/search/#rate-limit Because of that, I had to lower Gitlab::GithubImport::Client::RATE_LIMIT_THRESHOLD value from 50 to 10, because Gitlab::GithubImport::Client would rate limit itself after a call to search api. A default rate limit (when making a call to repositories endpoint, for example) is 5000.

Screenshots (strongly suggested)

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov

Merge request reports

Loading