Add per-request limit to parallel remote includes
What does this MR do and why?
After adding the parallel support of remote includes, we needed to limit the number of parallel HTTP requests to avoid abusing IO.
This change adds 2 as the maximum parallel limit per pipeline creation or lint request.
Related to #351250 (closed)
Implementation table: #351250 (comment 1647814064)
Screenshots or screen recordings
- Enable the feature flag;
Feature.enable(:ci_parallel_remote_includes)
- Run a basic web server to test slow includes.
require 'rack' def run(env) puts "URL: #{env['REQUEST_URI']} started at #{Time.now}" sleep 5 puts "URL: #{env['REQUEST_URI']} finished at #{Time.now}" " test: script: echo 'hello world' " end app = Proc.new do |env| [ 200, { "Content-Type" => "text/x-yaml" }, [run(env)] ] end Rack::Server.start(app: app, Port: 9292)
- Enable local includes on the Admin page http://gdk.test:3000/admin/application_settings/network
- Use a basic CI config;
include: - remote: http://0.0.0.0:9292/file1.yml - remote: http://0.0.0.0:9292/file2.yml - remote: http://0.0.0.0:9292/file3.yml - remote: http://0.0.0.0:9292/file4.yml - remote: http://0.0.0.0:9292/file5.yml build: script: exit 0
- Result;
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Furkan Ayhan