Add batch support while querying downloadable artifacts
What does this MR do and why?
I have a customer whose job was killed whenever they ran project.builds.with_downloadable_artifacts
. The solution is to query in small batches to reduce memory utilization.
I have seen several tickets with the same error.
- https://gitlab.zendesk.com/agent/tickets/420046
- https://gitlab.zendesk.com/agent/tickets/431049
- https://gitlab.zendesk.com/agent/tickets/429522
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
Run the following script in the rails console.
# Get debug output
Rails.logger.level = Logger::DEBUG
project = Project.find_by_full_path('path/to/project')
builds_with_artifacts = project.builds.with_downloadable_artifacts
After running the script, the job is killed
In the dmesg logs
Killed process 5648 (ruby) total-vm:5757928kB, anon-rss:3977872kB, file-rss:0kB, shmem-rss:0kB
After changing the script and adding batches, the scripts run without getting killed.
# Get debug output
Rails.logger.level = Logger::DEBUG
project = Project.find_by_full_path('path/to/project')
builds_with_artifacts = project.builds.with_downloadable_artifacts.find_each(batch_size: 1000)
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by David Wainaina