Skip to content

Add worker has external dependencies

Lucie Zhao requested to merge 363601_worker into master

What does this MR do and why?

Add worker dependency declarations to Sync workers and the Metrics worker for Geo. Because:

See https://docs.gitlab.com/ee/development/sidekiq/worker_attributes.html#jobs-with-external-dependencies.

I think it makes sense to use this on at least "sync jobs" because in that case, the secondary has no control over how long it will take to download the file or fetch the repo, and it can legitimately be on the order of hours.

Using worker_has_external_dependencies! will have the following effect:

  1. Most external dependencies (such as web-hooks) do not provide SLOs, and therefore we cannot guarantee the execution latencies on these jobs. Since we cannot guarantee execution latency, we cannot ensure throughput and therefore, in high-traffic environments, we need to ensure that jobs with external dependencies are separated from high urgency jobs, to ensure throughput on those queues.

  2. Errors in jobs with external dependencies have higher alerting thresholds as there is a likelihood that the cause of the error is external.

Resolves: #363601 (closed)

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Use Geo.
  2. Geo continues to sync and produce metrics as normal.
Edited by Michael Kozono

Merge request reports

Loading