Download export parts in parallel
What does this MR do and why?
This MR installs the async
gem which will allow us to download the export files concurrently using Fibers.
Other alternatives considered
- Using a thread pool: master...minac_use_multiple_threads_to_download_segment_files
- Implementing a fiber scheduler: master...minac_implement_non_blocking_file_download
I have decided to install the async gem because;
- It's a well-maintained gem.
- The thread pool approach does not look that clean.
- Implementing a fiber scheduler wasn't that hard as I copied and modified the test one from the Ruby repository but I didn't want to write specs and maintain it.
You can see the benefit of using the non-blocking IO feature of Ruby 3 with fibers by running the following in the rails console;
require 'async'
def test_async
start_time = Time.now.to_i
Async do
100.times do
Async do
HTTParty.get("https://httpbin.org/delay/2")
end
end
end
puts "Finished in #{Time.now.to_i - start_time} seconds"
end
test_async
Related to #440163 (closed)
Edited by Gregory Havenga