Filter GitHub projects to import using GitHub Search API
What does this MR do?
This MR replaces GitHub Importer's filtering functionality from filtering on GitLab server side to use GitHub Search API in order to implement pagination in the future MR (as well as being able to filter results and have them paginated).
There are a few caveats:
- In order to compose a correct search query that includes user's repositories as well as repos from organisations and from other users (where user is a collaborator) we need to do a few network requests to fetch usernames and org names.
- A list of repos where user is collaborator is fetched in order to retrieve a list of usernames owning those repos. I could not find a better way of fetching this list.
- A list of organisations is fetched to add to the overall search query
- GitHub search API rate limit is 30 per minute https://developer.github.com/v3/search/#rate-limit Because of that, I had to lower
Gitlab::GithubImport::Client::RATE_LIMIT_THRESHOLD
value from 50 to 10, becauseGitlab::GithubImport::Client
would rate limit itself after a call to search api. A default rate limit (when making a call to repositories endpoint, for example) is 5000.
Screenshots (strongly suggested)
Does this MR meet the acceptance criteria?
Conformity
-
Changelog entry - [-] Documentation (if required)
-
Code review guidelines - [-] Merge request performance guidelines
-
Style guides - [-] Database guides
- [-] Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team
Edited by George Koltsov