Optimize decryption of CI variables
What does this MR do and why?
In pipelines that have many environments and many CI variables, the
decryption in Ci::HasVariable#to_runner_variable
is a
significant bottleneck due to repeated decryption via
OpenSSL::PKCS5.pbkdf2_hmac
. We can optimize this by caching the
value in the request. In one customer's pipeline, this cuts down
Ci::CreatePipelineService
from 80+ seconds to 30 seconds.
Relates to #326271 (closed)
How to set up and validate locally
-
Enable
Feature.enable(:enable_ci_variable_caching)
. -
Create 170 environments and 100+ CI group variables (see #326271 (comment 812180988)) and profile the creation of the pipeline. For example:
project = Project.find(15)
current_user = User.first
pipeline_options = {}
pipeline_params = {
before: '3bb7f2ff983cf6ceb62be4408b0e651c2c605c70',
after: '11ca6d4aa27d17beeb49b63e0286e013fd149655',
ref: 'refs/heads/master',
checkout_sha: '11ca6d4aa27d17beeb49b63e0286e013fd149655'
}
service = Ci::CreatePipelineService.new(project, current_user, pipeline_params)
require 'stackprof'
RequestStore.begin!
Benchmark.measure do
StackProf.run(mode: :wall, out: '/tmp/create-pipeline-service-test4.dump', interval: 1000, raw: true) do
service.execute!(:push, pipeline_options)
end
end
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Edited by Stan Hu