Reduce variables transformations when creating a new pipeline (!73378) · Merge requests · GitLab.org / GitLab

Marius Bobin requested to merge ci-reduce-variables-objects-allocation-on-create into master Oct 29, 2021

What does this MR do and why?

Tries to reduce the number of Hash allocations when creating a new pipeline. The data was extracted using the gitlab-org/gitlab project with request_profiling and RAILS_PROFILE=true on my local env.

For this project we have around 800 jobs and a lot of rules to select what jobs can be created. For each rule statement we were transforming the variables from a Collection to a Hash.

Screenshots or screen recordings

Before

Total allocated: 1181462708 bytes (12207321 objects)
Total retained:  33618352 bytes (258826 objects)

...

allocated memory by file
-----------------------------------
 228457487  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/backtrace_cleaner.rb
 131114852  /Users/marius/.gem/ruby/2.7.4/gems/numerizer-0.2.0/lib/numerizer.rb
  92263425  /Users/marius/.gem/ruby/2.7.4/gems/marginalia-1.10.0/lib/marginalia/comment.rb
  69025560  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/ci/variables/collection/item.rb
  60540520  /Users/marius/.gem/ruby/2.7.4/gems/json-2.5.1/lib/json/common.rb
  51897774  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/json.rb
  39035008  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb
  34646760  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/ci/variables/collection.rb

...

allocated memory by location
-----------------------------------
 211872319  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/backtrace_cleaner.rb:96
  91041601  /Users/marius/.gem/ruby/2.7.4/gems/marginalia-1.10.0/lib/marginalia/comment.rb:115
  47388768  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/ci/variables/collection/item.rb:52

...

allocated objects by class
-----------------------------------
   5610708  String
   3342960  Array
   1528330  Hash
    367492  ActiveSupport::JSON::Encoding::JSONGemEncoder::EscapedString
    168922  Proc
    149545  Gitlab::Config::Entry::Factory
    148541  Gitlab::Ci::Variables::Collection::Item

After

Total allocated: 1094791849 bytes (11644219 objects)
Total retained:  30588247 bytes (238483 objects)

...

allocated memory by file
-----------------------------------
 224499662  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/backtrace_cleaner.rb
 131114852  /Users/marius/.gem/ruby/2.7.4/gems/numerizer-0.2.0/lib/numerizer.rb
  92224265  /Users/marius/.gem/ruby/2.7.4/gems/marginalia-1.10.0/lib/marginalia/comment.rb
  60253296  /Users/marius/.gem/ruby/2.7.4/gems/json-2.5.1/lib/json/common.rb
  51487041  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/json.rb
  38621912  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb
  37581208  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/ci/variables/collection/item.rb
  29846112  /Users/marius/Workspace/gdk/gitlab/lib/peek/views/active_record.rb
  29478505  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/json/encoding.rb
  25473992  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/core_ext/hash/keys.rb
  20335448  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/ci/variables/collection.rb

...

allocated memory by location
-----------------------------------
 208240222  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/backtrace_cleaner.rb:96
  91003564  /Users/marius/.gem/ruby/2.7.4/gems/marginalia-1.10.0/lib/marginalia/comment.rb:115
  47162220  /Users/marius/.gem/ruby/2.7.4/gems/numerizer-0.2.0/lib/numerizer.rb:126
  43884432  /Users/marius/.gem/ruby/2.7.4/gems/numerizer-0.2.0/lib/numerizer.rb:123
  36433872  /Users/marius/.gem/ruby/2.7.4/gems/json-2.5.1/lib/json/common.rb:216
  29390328  /Users/marius/Workspace/gdk/gitlab/lib/peek/views/active_record.rb:64
  23165712  /Users/marius/.gem/ruby/2.7.4/gems/json-2.5.1/lib/json/common.rb:312
  19842297  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/core_ext/object/json.rb:40
  18591880  /Users/marius/.gem/ruby/2.7.4/gems/activesupport-6.1.4.1/lib/active_support/callbacks.rb:409
  17838144  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/utils/strong_memoize.rb:45
  15969912  /Users/marius/Workspace/gdk/gitlab/lib/gitlab/ci/variables/collection/item.rb:52

...

allocated objects by class
-----------------------------------
   5498828  String
   3111750  Array
   1327279  Hash
    364801  ActiveSupport::JSON::Encoding::JSONGemEncoder::EscapedString
    171545  Proc
    149538  Gitlab::Config::Entry::Factory
    148374  Gitlab::Ci::Variables::Collection::Item

How to set up and validate locally

import gitlab-org/gitlab locally
export RAILS_PROFILE=true
open dev tools, network tab
create a new pipeline
look for the POST request in the network tab, select edit and resend, add the profiling headers and hit send

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

I have evaluated the MR acceptance checklist for this MR.

Related to #322386 (closed)

Edited Nov 05, 2021 by Marius Bobin

Reduce variables transformations when creating a new pipeline