Use 1.week TTL for importer JobWaiter keys
What does this MR do and why?
This MR extends the JobWaiter
key expiry time for importers to 1 week.
Click to read a description of `JobWaiter` and how it's used in importers.
- Our importers will queue a number of workers to process data fetched from APIs like GitHub. For each worker queued we increment a
JobWaiter
counter to track the count of jobs queued. - When one of the queued worker completes, it calls
JobWaiter.notify
to report that it has finished. This pushes its Sidekiq Job ID (jid
) to a Redis set. (What's pushed isn't important but rather just that 1 of something was pushed to the set). -
Gitlab::GithubImport::AdvanceStageWorker
is queued with the count of jobs queued. It callsJobWaiter.wait
to clear the redis set and decrement the count. It does this periodically until the count is 0 (when all jobs have been notified that they completed).
In very large imports sometimes more than 6 hours (the current key expiry time) pass between when the last job of a certain batch of jobs notifies (calls JobWaiter.notify
), and when we read the job waiter set (call JobWaiter.wait
).
This results in these imports becoming stuck, as they continue to wait for the expired notifications.
Extending this time to 7 days will cover very large imports. Shorter imports will continue to pop data from the set as soon as they call JobWaiter.wait
, so for non-stuck imports (the vast majority) this change won't increase the size of data stored in Redis.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #422976 (closed)