Skip to content

Cache Fog::Storage object for CI build chunks

Stan Hu requested to merge sh-cache-build-chunks-fog into master

What does this MR do and why?

When CI build chunks are processed with object storage, previously a new Fog::Storage instance was created each time the chunk had to be read or deleted. This led to new Google HTTPS client to be created each time this occurred, which causes a surprisingly high amount of overhead since the client has to set up SSL and request credentials from the instance metadata endpoint.

We can reduce this overhead by caching Fog::Storage with RequestStore so the job can reuse this across mutiple operations.

Relates to #487908

How to set up and validate locally

  1. On some Google VM, set up object storage using Google Application Credentials.
  2. Run Feature.enable(:ci_enable_live_trace) in the gitlab-rails console.
  3. Apply this patch.
  4. Start up a job that generates a lot of logs:
builds:
  script:
    - |
      for i in {1..10000}; do
        echo "Build ${CI_NODE_INDEX} - Line $i: Lorem ipsum dolor sit amet, consectetur adipiscing elit.";
      done
  1. Job logs should be archived properly.
Edited by Stan Hu

Merge request reports

Loading