Skip to content

Support worker_name predicate in Sidekiq queue selector

What does this MR do?

Closes gitlab-com/gl-infra/scalability#1018 (closed)

We are supporting queue selector as an option for the sidekiq cluster to target a set of particular queues (actually workers) based on an input set of selector. In the current initiative to reduce the number of watching queues, it turns out we need to support worker_name predicate so that we can target a precise worker during rolling out the new worker routing feature.

Recently, after !59550 (merged) is merged, the worker matching logic is consolidated into Gitlab::SidekiqConfig::WorkerMatcher. The input of this matcher is the worker metadata generated from Gitlab::SidekiqConfig::Worker. The reason for this is that the queue selecting matching is shared between two components:

  • The queue router (gitlab-com/gl-infra/scalability#1016 (closed))
  • The Sidekiq cluster CLI. When the script starts, it loads all the worker information from app/workers/all_queues.yml and ee/app/workers/all_queues.yml, uses the matcher to filter out target queues, and actually starts the Sidekiq processes. If we use a worker class as an input of the matcher, all the workers must be loaded. The starting time of the CLI is likely to be double.

Hence, this MR implements the support of worker_name by adding the worker_name to the mentioned worker metadata, and tweak the predicate list of the worker matcher. As a result, that makes the all_queues.yml files changed as well. The newly added attribute is just a minor additional information. It doesn't affect the main logic of queue selector.

Screenshots (strongly suggested)

  • Use worker_name to target some precise workers

Screen_Shot_2021-04-23_at_12.50.30

  • Other predicates are still working as expected

Screen_Shot_2021-04-23_at_12.50.01

Screen_Shot_2021-04-23_at_12.50.12

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by Quang-Minh Nguyen

Merge request reports

Loading