Improve performance of related branches finder
What does this MR do and why?
Improves memory consumption of the related branches finder by searching using Gitaly instead of loading all the branches and filtering in Ruby. This also saves us extra Gitaly calls to get the SHA of the matching branches.
Loading all the branches from Redis (because this is cached in Redis) is expensive for projects with large number of branches. I tested this locally on a project with 5k branches and ran a memory profile.
Before:
Total allocated: 66859038 bytes (67289 objects)
Total retained: 1620453 bytes (5150 objects)
allocated memory by gem
-----------------------------------
64213008 redis-4.4.0
2166666 bootsnap-1.12.0
365541 other
46995 gitlab/lib
25599 activesupport-6.1.4.7
18131 marginalia-1.10.0
12009 activerecord-6.1.4.7
3360 connection_pool-2.2.5
1560 flipper-0.21.0
1290 timecop-0.9.1
1088 prometheus-client-mmap-0.15.0
793 gitlab/app
765 zeitwerk-2.6.0
704 activemodel-6.1.4.7
576 sentry-ruby-core-5.1.1
352 flipper-active_support_cache_store-0.21.0
168 bullet-6.1.3
160 set-1.0.1
153 lib
120 grape_logging-1.8.4
After:
Total allocated: 693205 bytes (4649 objects)
Total retained: 190865 bytes (415 objects)
allocated memory by gem
-----------------------------------
389130 activesupport-6.1.4.7
181567 activerecord-6.1.4.7
46643 other
23069 marginalia-1.10.0
22016 pg-1.3.5
14160 lib
6976 gitlab/lib
3045 grpc-1.42.0
1567 redis-4.4.0
1118 timecop-0.9.1
1005 gitlab/app
840 connection_pool-2.2.5
624 activemodel-6.1.4.7
496 prometheus-client-mmap-0.15.0
317 fast_gettext-2.1.0
208 bullet-6.1.3
160 set-1.0.1
144 sentry-ruby-core-5.1.1
120 grape_logging-1.8.4
I also checked the before / after numbers related to request duration:
Project with 73 branches
- Before
{"redis_calls":8,"redis_duration_s":0.003367,"redis_read_bytes":2045,"redis_write_bytes":77340,"redis_cache_calls":2,"redis_cache_duration_s":0.002006, "redis_cache_read_bytes":1795,"redis_cache_write_bytes":76048,"db_duration_s":0.01573,"view_duration_s":0.0001,"duration_s":0.15106}
- After
{"gitaly_calls":1,"gitaly_duration_s":0.008692,"redis_calls":8,"redis_duration_s":0.001709,"redis_read_bytes":360,"redis_write_bytes":81420,"redis_cache_calls":2,"redis_cache_duration_s":0.000855, "redis_cache_read_bytes":110,"redis_cache_write_bytes":80128,"db_duration_s":0.0112,"view_duration_s":0.0001,"duration_s":0.12981}
Project with 5,074 branches
- Before
{"redis_calls":8,"redis_duration_s":0.005205,"redis_read_bytes":218493,"redis_write_bytes":23857,"redis_cache_calls":2,"redis_cache_duration_s":0.004177, "redis_cache_read_bytes":218235,"redis_cache_write_bytes":22564,"db_duration_s":0.0084,"view_duration_s":0.0001,"duration_s":0.40696}
- After
{"gitaly_calls":1,"gitaly_duration_s":0.009424,"redis_calls":8,"redis_duration_s":0.0016870000000000001,"redis_read_bytes":368,"redis_write_bytes":25204,"redis_cache_calls":2,"redis_cache_duration_s":0.000441, "redis_cache_read_bytes":110,"redis_cache_write_bytes":23910,"db_duration_s":0.00391,"view_duration_s":0.00013,"duration_s":0.05679}
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.