Handle a slow query from CodeNavigationPath
What does this MR do and why?
Contributes to #472420 (closed)
Problem
pipeline.job_artifacts.with_file_types(['lsif']).last
can trigger an
inefficient query that blocks the blob page loading request.
Solution
- Add a timeout around the query to limit the execution time.
- Use a curcuit breaker to disable CodeNavigation feature for 24 hours when artifacts cannot be loaded.
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
How to set up and validate locally
- Patch the code to emulate a long running query
latest_commits_shas =
project.repository.commits(commit_sha, limit: LATEST_COMMITS_LIMIT).map(&:sha)
- pipeline = @project.ci_pipelines.for_sha(latest_commits_shas).last
+ # pipeline = @project.ci_pipelines.for_sha(latest_commits_shas).last
- next unless pipeline
+ # next unless pipeline
artifact = Timeout.timeout(ARTIFACT_TIMEOUT) do
- pipeline.job_artifacts.with_file_types(['lsif']).last
+ sleep 15
+ # pipeline.job_artifacts.with_file_types(['lsif']).last
end
artifact&.job
- Visit any blob view page (for example, http://gdk.test:3000/gitlab-org/gitlab-test/-/blob/master/README?ref_type=heads)
- You should notice that it takes >10 seconds to load.
- Refresh the page a few times
- After 3-5 refreshes, the page should load much faster (it means that the circuit breaker is open)
To confirm circuit breaker status, open Rails console
Gitlab::Redis::RateLimiting.with { |a| a }.keys
=> ["circuits:CodeNavigationPath:open", "circuits:CodeNavigationPath:stats:1723809600:failure", "circuits:CodeNavigationPath:half_open"]
-
circuits:CodeNavigationPath:open
means that the circuit breaker is open. -
circuits:CodeNavigationPath:stats:1723809600:failure
contains number of failures
Gitlab::Redis::RateLimiting.with { |a| a }.get("circuits:CodeNavigationPath:stats:1723809600:failure")
=> "7"
Gitlab::Redis::RateLimiting.with { |a| a }.ttl("circuits:CodeNavigationPath:open")
=> 86116 # ~24 hours
Edited by Vasilii Iakliushin