Suggested reviewer project registration worker

What does this MR do and why?

Suggested reviewer project registration worker

• Behind a feature flag, to be tested by development Related to #373013 (closed)

Screenshots or screen recordings

1. Enable the suggested reviewers, after a disabled state

2. Check the logs

$ grep -r RegisterSuggestedReviewersProjectWorker log/sidekiq*

log/sidekiq.log:{"severity":"INFO","time":"2022-09-30T09:45:46.696Z","retry":25,"queue":"default","backtrace":true,"version":0,"args":["20","46"],"class":"Projects::RegisterSuggestedReviewersProjectWorker","jid":"a7e842e5d75bde7d8df1d172","created_at":"2022-09-30T09:45:46.683Z","correlation_id":"01GE6WK9FP31GKXW33NZMVKE8Z","meta.caller_id":"Projects::Settings::MergeRequestsController#update","meta.remote_ip":"172.16.123.1","meta.feature_category":"workflow_automation","meta.user":"test1","meta.project":"zama/zingo","meta.root_namespace":"zama","meta.client_id":"user/46","meta.root_caller_id":"Projects::Settings::MergeRequestsController#update","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:add372d2b713cd4fce99abc2970f7cd89b681894f611a39fe0991daf0395a9ed","size_limiter":"validated","enqueued_at":"2022-09-30T09:45:46.695Z","job_size_bytes":7,"pid":1749109,"message":"Projects::RegisterSuggestedReviewersProjectWorker JID-a7e842e5d75bde7d8df1d172: start","job_status":"start","scheduling_latency_s":0.001494}
log/sidekiq.log:{"severity":"INFO","time":"2022-09-30T09:45:47.314Z","retry":25,"queue":"default","backtrace":true,"version":0,"args":[],"class":"Namespaces::ProcessSyncEventsWorker","jid":"8a5d25d48cae77571dbdeb01","created_at":"2022-09-30T09:45:47.308Z","correlation_id":"01GE6WK9FP31GKXW33NZMVKE8Z","meta.caller_id":"Projects::RegisterSuggestedReviewersProjectWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"pods","meta.user":"test1","meta.project":"zama/zingo","meta.root_namespace":"zama","meta.client_id":"user/46","meta.root_caller_id":"Projects::Settings::MergeRequestsController#update","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:99b01e5f7dc2a9b47ce31029252a21c1a7e02ef97b2a00a70d65582147f34ddb","size_limiter":"validated","enqueued_at":"2022-09-30T09:45:47.310Z","job_size_bytes":2,"pid":1749109,"message":"Namespaces::ProcessSyncEventsWorker JID-8a5d25d48cae77571dbdeb01: start","job_status":"start","scheduling_latency_s":0.004024}
log/sidekiq.log:{"severity":"INFO","time":"2022-09-30T09:45:47.473Z","retry":25,"queue":"default","backtrace":true,"version":0,"queue_namespace":"authorized_project_update","class":"AuthorizedProjectUpdate::ProjectRecalculatePerUserWorker","args":["20","111"],"jid":"d8bf91ff668f2627d16a57db","created_at":"2022-09-30T09:45:47.468Z","correlation_id":"01GE6WK9FP31GKXW33NZMVKE8Z","meta.caller_id":"Projects::RegisterSuggestedReviewersProjectWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"authentication_and_authorization","meta.user":"test1","meta.project":"zama/zingo","meta.root_namespace":"zama","meta.client_id":"user/46","meta.root_caller_id":"Projects::Settings::MergeRequestsController#update","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:bf2e5d5ae6010e600704e60636e3758e4cffb168076bf9a37fce65ba65a2caf1","size_limiter":"validated","enqueued_at":"2022-09-30T09:45:47.470Z","job_size_bytes":8,"pid":1749109,"message":"AuthorizedProjectUpdate::ProjectRecalculatePerUserWorker JID-d8bf91ff668f2627d16a57db: start","job_status":"start","scheduling_latency_s":0.003206}


log/sidekiq.log:{"severity":"WARN","time":"2022-09-30T09:45:47.576Z","retry":25,"queue":"default","backtrace":true,"version":0,"args":["20","46"],"class":"Projects::RegisterSuggestedReviewersProjectWorker","jid":"a7e842e5d75bde7d8df1d172","created_at":"2022-09-30T09:45:46.683Z","correlation_id":"01GE6WK9FP31GKXW33NZMVKE8Z","meta.caller_id":"Projects::Settings::MergeRequestsController#update","meta.remote_ip":"172.16.123.1","meta.feature_category":"workflow_automation","meta.user":"test1","meta.project":"zama/zingo","meta.root_namespace":"zama","meta.client_id":"user/46","meta.root_caller_id":"Projects::Settings::MergeRequestsController#update","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:add372d2b713cd4fce99abc2970f7cd89b681894f611a39fe0991daf0395a9ed","size_limiter":"validated","enqueued_at":"2022-09-30T09:45:46.695Z","job_size_bytes":7,"pid":1749109,"message":"Projects::RegisterSuggestedReviewersProjectWorker JID-a7e842e5d75bde7d8df1d172: fail: 0.879816 sec","job_status":"fail","scheduling_latency_s":0.001494,"redis_calls":41,"redis_duration_s":0.011408999999999999,"redis_read_bytes":2895,"redis_write_bytes":5761,"redis_cache_calls":18,"redis_cache_duration_s":0.005801,"redis_cache_read_bytes":2758,"redis_cache_write_bytes":1177,"redis_queues_calls":21,"redis_queues_duration_s":0.004696,"redis_queues_read_bytes":135,"redis_queues_write_bytes":4400,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000912,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":184,"db_count":112,"db_write_count":29,"db_cached_count":0,"db_replica_count":0,"db_primary_count":112,"db_main_count":112,"db_ci_count":0,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.328,"db_main_duration_s":0.328,"db_ci_duration_s":0.0,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.613567,"worker_id":"sidekiq_0","rate_limiting_gates":[],"duration_s":0.879816,"completed_at":"2022-09-30T09:45:47.576Z","load_balancing_strategy":"primary","exception.class":"Gitlab::AppliedMl::Errors::ConfigurationError","exception.message":"Variable SUGGESTED_REVIEWERS_SECRET is missing","exception.backtrace":["ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:80:in `read_secret!'","ee/lib/gitlab/applied_ml/suggested_reviewers/client.rb:29:in `initialize'","ee/app/services/projects/register_suggested_reviewers_project_service.rb:14:in `new'","ee/app/services/projects/register_suggested_reviewers_project_service.rb:14:in `execute'","ee/app/workers/projects/register_suggested_reviewers_project_worker.rb:24:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/strategies/until_executed.rb:17:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/duplicate_job.rb:58:in `perform'","lib/gitlab/sidekiq_middleware/duplicate_jobs/server.rb:8:in `call'","lib/gitlab/sidekiq_middleware/worker_context.rb:9:in `wrap_in_optional_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:19:in `block in call'","lib/gitlab/application_context.rb:113:in `block in use'","lib/gitlab/application_context.rb:113:in `use'","lib/gitlab/application_context.rb:54:in `with_context'","lib/gitlab/sidekiq_middleware/worker_context/server.rb:17:in `call'","lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'","lib/gitlab/sidekiq_versioning/middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `block in call'","lib/gitlab/database/query_analyzer.rb:37:in `within'","lib/gitlab/sidekiq_middleware/query_analyzer.rb:7:in `call'","lib/gitlab/sidekiq_middleware/admin_mode/server.rb:14:in `call'","lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:9:in `call'","lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'","lib/gitlab/sidekiq_middleware/extra_done_log_metadata.rb:7:in `call'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:10:in `block in call'","lib/gitlab/with_request_store.rb:17:in `enabling_request_store'","lib/gitlab/with_request_store.rb:10:in `with_request_store'","lib/gitlab/sidekiq_middleware/request_store_middleware.rb:9:in `call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:76:in `block in call'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `block in instrument'","lib/gitlab/metrics/background_transaction.rb:33:in `run'","lib/gitlab/sidekiq_middleware/server_metrics.rb:103:in `instrument'","lib/gitlab/sidekiq_middleware/server_metrics.rb:75:in `call'","lib/gitlab/sidekiq_middleware/monitor.rb:10:in `block in call'","lib/gitlab/sidekiq_daemon/monitor.rb:49:in `within_job'","lib/gitlab/sidekiq_middleware/monitor.rb:9:in `call'","lib/gitlab/sidekiq_middleware/size_limiter/server.rb:13:in `call'","lib/gitlab/sidekiq_logging/structured_logger.rb:21:in `call'"],"db_duration_s":0.11236}
log/sidekiq.log:{"severity":"INFO","time":"2022-09-30T09:45:47.917Z","retry":25,"queue":"default","backtrace":true,"version":0,"args":[],"class":"Namespaces::ProcessSyncEventsWorker","jid":"8a5d25d48cae77571dbdeb01","created_at":"2022-09-30T09:45:47.308Z","correlation_id":"01GE6WK9FP31GKXW33NZMVKE8Z","meta.caller_id":"Projects::RegisterSuggestedReviewersProjectWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"pods","meta.user":"test1","meta.project":"zama/zingo","meta.root_namespace":"zama","meta.client_id":"user/46","meta.root_caller_id":"Projects::Settings::MergeRequestsController#update","worker_data_consistency":"always","idempotency_key":"resque:gitlab:duplicate:default:99b01e5f7dc2a9b47ce31029252a21c1a7e02ef97b2a00a70d65582147f34ddb","size_limiter":"validated","enqueued_at":"2022-09-30T09:45:47.310Z","job_size_bytes":2,"pid":1749109,"message":"Namespaces::ProcessSyncEventsWorker JID-8a5d25d48cae77571dbdeb01: done: 0.602539 sec","job_status":"done","scheduling_latency_s":0.004024,"redis_calls":10,"redis_duration_s":0.0025250000000000003,"redis_read_bytes":620,"redis_write_bytes":1017,"redis_cache_calls":3,"redis_cache_duration_s":0.0008,"redis_cache_read_bytes":608,"redis_cache_write_bytes":234,"redis_queues_calls":5,"redis_queues_duration_s":0.001074,"redis_queues_read_bytes":10,"redis_queues_write_bytes":442,"redis_shared_state_calls":2,"redis_shared_state_duration_s":0.000651,"redis_shared_state_read_bytes":2,"redis_shared_state_write_bytes":341,"db_count":7,"db_write_count":2,"db_cached_count":0,"db_replica_count":0,"db_primary_count":7,"db_main_count":6,"db_ci_count":1,"db_main_replica_count":0,"db_ci_replica_count":0,"db_replica_cached_count":0,"db_primary_cached_count":0,"db_main_cached_count":0,"db_ci_cached_count":0,"db_main_replica_cached_count":0,"db_ci_replica_cached_count":0,"db_replica_wal_count":0,"db_primary_wal_count":0,"db_main_wal_count":0,"db_ci_wal_count":0,"db_main_replica_wal_count":0,"db_ci_replica_wal_count":0,"db_replica_wal_cached_count":0,"db_primary_wal_cached_count":0,"db_main_wal_cached_count":0,"db_ci_wal_cached_count":0,"db_main_replica_wal_cached_count":0,"db_ci_replica_wal_cached_count":0,"db_replica_duration_s":0.0,"db_primary_duration_s":0.017,"db_main_duration_s":0.015,"db_ci_duration_s":0.002,"db_main_replica_duration_s":0.0,"db_ci_replica_duration_s":0.0,"cpu_s":0.035733,"worker_id":"sidekiq_0","rate_limiting_gates":[],"extra.namespaces_process_sync_events_worker.estimated_total_events":1,"extra.namespaces_process_sync_events_worker.consumable_events":1,"extra.namespaces_process_sync_events_worker.processed_events":1,"duration_s":0.602539,"completed_at":"2022-09-30T09:45:47.917Z","load_balancing_strategy":"primary","db_duration_s":0.004471}

How to set up and validate locally

1. Ensure a SaaS (Gitlab.com) environment
   • One way of doing this is to add a env.runit file to the root GDK folder with the following snippet export GITLAB_SIMULATE_SAAS=1
2. Set ultimate license on a group http://gdk.test:3000/admin/groups
3. Create a project in the ultimate group
4. Set the feature flag on rails console bin/rails c

project = Project.find(id-of-your-project) in the rails console
Feature.enable(:suggested_reviewers_control, project)

1. Go to the merge request settings for the project
2. Enable the suggested reviewers, after a disabled state
3. You should see some failed workers in the sidekiq logs (due to lack of authentication) and some failed jobs in admin panel http://gdk.test:3000/admin/background_jobs

grep -r RegisterSuggestedReviewersProjectWorker log/sidekiq*

Flow

To help with my understanding, I have created a sequence diagram. Hopefully, other reviewers might find it useful.