Cache Fog::Storage object for CI build chunks
What does this MR do and why?
When CI build chunks are processed with object storage, previously a
new Fog::Storage
instance was created each time the chunk had to be
read or deleted. This led to new Google HTTPS client to be created
each time this occurred, which causes a surprisingly high amount of
overhead since the client has to set up SSL and request credentials
from the instance metadata endpoint.
We can reduce this overhead by caching Fog::Storage
with
RequestStore so the job can reuse this across mutiple operations.
Relates to #487908
How to set up and validate locally
- On some Google VM, set up object storage using Google Application Credentials.
- Run
Feature.enable(:ci_enable_live_trace)
in thegitlab-rails console
. - Apply this patch.
- Start up a job that generates a lot of logs:
builds:
script:
- |
for i in {1..10000}; do
echo "Build ${CI_NODE_INDEX} - Line $i: Lorem ipsum dolor sit amet, consectetur adipiscing elit.";
done
- Job logs should be archived properly.
Edited by Stan Hu