Skip to content

Optimize memory usage for user mapping in Direct Transfer jobs

Rodrigo Tomonari requested to merge rodrigo/improved-user-mapping-memory-usage-improvements into master

This commit reduces memory consumption during Direct Transfer jobs when improved user mapping is enabled by making several key adjustments:

  • Update UpdateSourceUsersService to not caching SQL queries
  • Increase max_retries_after_interruption for long-running jobs that may face multiple interruptions. Note that this can be done because Direct Transfer jobs continue from where they left off.
  • Update MemberPipeline so that source users are not cached, as the pipeline does not use them later.

Additionally, this reduces the LRU_CACHE_SIZE to 100. Testing multiple migrations revealed that a larger cache size consumes significant memory, especially when multiple jobs run concurrently. Each job keeps a separate in-memory cache, leading to excessive memory use.

Related to: #467512 (comment 2147214017)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Sidekiq memory usage when running 20 batched pipelines concurrently:

Screenshot_2024-10-07_at_19.47.03

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

Edited by Rodrigo Tomonari

Merge request reports