Skip to content

Add logging to the `WebHooks::AutoDisabling` concern

Rémy Coutable requested to merge add-logging-to-webhook-auto-disabling into master

What does this MR do and why?

  1. Add a Gitlab::WebHooks::Logger class
  2. Add logging to the WebHooks::AutoDisabling concern

This logging will help the EP team investigate auto-disabling of the gitlab-org group webhook (context: gitlab-org/quality/triage-ops#1373).

Screenshots or screen recordings

{"severity":"INFO","time":"2023-08-01T16:08:32.206Z","correlation_id":"01H6RXP8NEAZDMZTZRGNGBFBN1","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":1}
{"severity":"INFO","time":"2023-08-01T16:08:32.304Z","correlation_id":"01H6RXPBAB7ZJZ52NB8Q01BJ4B","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":2}
{"severity":"INFO","time":"2023-08-01T16:09:26.468Z","correlation_id":"01H6RXRZB6EMH23NZYHGYAS14P","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":3}
{"severity":"INFO","time":"2023-08-01T16:09:30.248Z","correlation_id":"01H6RXS33VWV16AYFC5PNHB300","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":4,"backoff_count":1,"disabled_until":"2023-08-01T16:10:30.248Z"}
{"severity":"INFO","time":"2023-08-01T16:11:01.354Z","correlation_id":"01H6RXVWX3TZJVJW4R6VYKZ6S3","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":5,"backoff_count":2,"disabled_until":"2023-08-01T16:13:01.354Z"}
{"severity":"INFO","time":"2023-08-01T16:14:48.878Z","correlation_id":"01H6RY2TRMK7D606QZ1B2ENVMD","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":6,"backoff_count":3,"disabled_until":"2023-08-01T16:18:48.878Z"}
{"severity":"INFO","time":"2023-08-01T16:22:50.081Z","correlation_id":"01H6RYCJBJ0TYHXM6WT0N5TW2Z","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":7,"backoff_count":4,"disabled_until":"2023-08-01T16:30:50.080Z"}
{"severity":"INFO","time":"2023-08-01T16:22:50.336Z","correlation_id":"01H6RYDWT7GJ5D4C0MEJJTE768","meta.caller_id":"WebHooks::LogExecutionWorker","meta.remote_ip":"172.16.123.1","meta.feature_category":"webhooks","meta.user":"root","meta.user_id":1,"meta.project":"gitlab-org/gitlab-shell","meta.root_namespace":"gitlab-org","meta.client_id":"user/1","meta.root_caller_id":"Projects::NotesController#create","meta.related_class":"GroupHook","meta.subscription_plan":"default","hook_id":1,"action":"backoff","recent_failures":8,"backoff_count":5,"disabled_until":"2023-08-01T16:38:50.335Z"}

How to set up and validate locally

  1. Create a group webhook for the local gitlab-org group that points to https://httpstat.us/500
  2. Tail the log/web_hooks.log log file: tail -f log/web_hooks.log
  3. Create issues or issue notes and see log lines being added

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Rémy Coutable

Merge request reports