Allow scheduling jobs in batch with contexts
What does this MR do?
This adds helper methods for the bulk_perform_in
and
bulk_perform_async
methods that allow specifying a context per job.
Scheduling can be done by creating a hash that maps worker arguments to the context to be used for those arguments:
{ [<arguments for worker>] => { user: user, project: project} }
These will be stored on the worker class when scheduling the job, so the sidekiq middleware can fetch the right context based on those arguments and apply them to the job when scheduling it.
This also implements this method for the UpdateAllMirrorsWorker
which schedules ProjectImportScheduleWorker
in batch. When adding
this, the ProjectImportScheduleWorker
-jobs will have a context.
Which means that job now shows up in the logs as follows:
2020-01-28_12:33:35.46230 rails-background-jobs : {"severity":"INFO","time":"2020-01-28T12:33:35.461Z","class":"ProjectImportScheduleWorker","args":[13],"retry":false,"queue":"project_import_schedule","backtrace":true,"jid":"535f30fd3bde0bcd8a74811d","created_at":"2020-01-28T12:33:27.754Z","enqueued_at":"2020-01-28T12:33:28.312Z","meta.project":"gitlab-org/gitlab","meta.root_namespace":"gitlab-org","meta.subscription_plan":"default","meta.caller_id":"UpdateAllMirrorsWorker","correlation_id":"PJHJKnbQdL1","pid":50331,"message":"ProjectImportScheduleWorker JID-535f30fd3bde0bcd8a74811d: done: 1.294023 sec","job_status":"done","scheduling_latency_s":5.855765,"duration":1.294023,"cpu_s":0.27924,"completed_at":"2020-01-28T12:33:35.461Z","db_duration":7.35100032761693,"db_duration_s":0.00735100032761693}
gitlab-com/gl-infra/scalability#112 (closed)
Some parts are duplicated with !23444 (merged)
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation (if required) Documentation will be part of gitlab-com/gl-infra/scalability#113 (closed)
-
Code review guidelines -
Merge request performance guidelines -
Style guides - [-] Database guides
-
Separation of EE specific content