Skip to content

Improve performance of related branches finder

Heinrich Lee Yu requested to merge related-branches-list-refs into master

What does this MR do and why?

Improves memory consumption of the related branches finder by searching using Gitaly instead of loading all the branches and filtering in Ruby. This also saves us extra Gitaly calls to get the SHA of the matching branches.

Loading all the branches from Redis (because this is cached in Redis) is expensive for projects with large number of branches. I tested this locally on a project with 5k branches and ran a memory profile.

Before:

Total allocated: 66859038 bytes (67289 objects)
Total retained:  1620453 bytes (5150 objects)

allocated memory by gem
-----------------------------------
  64213008  redis-4.4.0
   2166666  bootsnap-1.12.0
    365541  other
     46995  gitlab/lib
     25599  activesupport-6.1.4.7
     18131  marginalia-1.10.0
     12009  activerecord-6.1.4.7
      3360  connection_pool-2.2.5
      1560  flipper-0.21.0
      1290  timecop-0.9.1
      1088  prometheus-client-mmap-0.15.0
       793  gitlab/app
       765  zeitwerk-2.6.0
       704  activemodel-6.1.4.7
       576  sentry-ruby-core-5.1.1
       352  flipper-active_support_cache_store-0.21.0
       168  bullet-6.1.3
       160  set-1.0.1
       153  lib
       120  grape_logging-1.8.4

After:

Total allocated: 693205 bytes (4649 objects)
Total retained:  190865 bytes (415 objects)

allocated memory by gem
-----------------------------------
    389130  activesupport-6.1.4.7
    181567  activerecord-6.1.4.7
     46643  other
     23069  marginalia-1.10.0
     22016  pg-1.3.5
     14160  lib
      6976  gitlab/lib
      3045  grpc-1.42.0
      1567  redis-4.4.0
      1118  timecop-0.9.1
      1005  gitlab/app
       840  connection_pool-2.2.5
       624  activemodel-6.1.4.7
       496  prometheus-client-mmap-0.15.0
       317  fast_gettext-2.1.0
       208  bullet-6.1.3
       160  set-1.0.1
       144  sentry-ruby-core-5.1.1
       120  grape_logging-1.8.4

I also checked the before / after numbers related to request duration:

Project with 73 branches

  1. Before

{"redis_calls":8,"redis_duration_s":0.003367,"redis_read_bytes":2045,"redis_write_bytes":77340,"redis_cache_calls":2,"redis_cache_duration_s":0.002006, "redis_cache_read_bytes":1795,"redis_cache_write_bytes":76048,"db_duration_s":0.01573,"view_duration_s":0.0001,"duration_s":0.15106}

  1. After

{"gitaly_calls":1,"gitaly_duration_s":0.008692,"redis_calls":8,"redis_duration_s":0.001709,"redis_read_bytes":360,"redis_write_bytes":81420,"redis_cache_calls":2,"redis_cache_duration_s":0.000855, "redis_cache_read_bytes":110,"redis_cache_write_bytes":80128,"db_duration_s":0.0112,"view_duration_s":0.0001,"duration_s":0.12981}

Project with 5,074 branches

  1. Before

{"redis_calls":8,"redis_duration_s":0.005205,"redis_read_bytes":218493,"redis_write_bytes":23857,"redis_cache_calls":2,"redis_cache_duration_s":0.004177, "redis_cache_read_bytes":218235,"redis_cache_write_bytes":22564,"db_duration_s":0.0084,"view_duration_s":0.0001,"duration_s":0.40696}

  1. After

{"gitaly_calls":1,"gitaly_duration_s":0.009424,"redis_calls":8,"redis_duration_s":0.0016870000000000001,"redis_read_bytes":368,"redis_write_bytes":25204,"redis_cache_calls":2,"redis_cache_duration_s":0.000441, "redis_cache_read_bytes":110,"redis_cache_write_bytes":23910,"db_duration_s":0.00391,"view_duration_s":0.00013,"duration_s":0.05679}

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Heinrich Lee Yu

Merge request reports

Loading