Support worker_name predicate in Sidekiq queue selector
What does this MR do?
Closes gitlab-com/gl-infra/scalability#1018 (closed)
We are supporting queue selector as an option for the sidekiq cluster to target a set of particular queues (actually workers) based on an input set of selector. In the current initiative to reduce the number of watching queues, it turns out we need to support worker_name
predicate so that we can target a precise worker during rolling out the new worker routing feature.
Recently, after !59550 (merged) is merged, the worker matching logic is consolidated into Gitlab::SidekiqConfig::WorkerMatcher
. The input of this matcher is the worker metadata generated from Gitlab::SidekiqConfig::Worker
. The reason for this is that the queue selecting matching is shared between two components:
- The queue router (gitlab-com/gl-infra/scalability#1016 (closed))
- The Sidekiq cluster CLI. When the script starts, it loads all the worker information from
app/workers/all_queues.yml
andee/app/workers/all_queues.yml
, uses the matcher to filter out target queues, and actually starts the Sidekiq processes. If we use a worker class as an input of the matcher, all the workers must be loaded. The starting time of the CLI is likely to be double.
Hence, this MR implements the support of worker_name
by adding the worker_name
to the mentioned worker metadata, and tweak the predicate list of the worker matcher. As a result, that makes the all_queues.yml
files changed as well. The newly added attribute is just a minor additional information. It doesn't affect the main logic of queue selector.
Screenshots (strongly suggested)
- Use
worker_name
to target some precise workers
- Other predicates are still working as expected
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. -
I have not included a changelog entry because _____.
-
-
Documentation (if required) -
Code review guidelines -
Merge request performance guidelines -
Style guides -
Database guides -
Separation of EE specific content
Availability and Testing
-
Review and add/update tests for this feature/bug. Consider all test levels. See the Test Planning Process. -
Tested in all supported browsers -
Informed Infrastructure department of a default or new setting change, if applicable per definition of done
Security
If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:
-
Label as security and @ mention @gitlab-com/gl-security/appsec
-
The MR includes necessary changes to maintain consistency between UI, API, email, or other methods -
Security reports checked/validated by a reviewer from the AppSec team