Handle some WebHookService errors
What does this MR do?
WebHookWorker
is currently experiencing ~12k unhandled errors per week.
The errors can be seen in Kibana of counts over the past 7 days:
This MR handles the following errors:
EOFError
EOFError
can be due to using port 443
without SSL, or transient network failures.
The SSL configuration problem is unlikely because HTTParty
handles setting this, so it's likely this error is due to the remote server terminating the connection.
ActiveRecord::RecordNotFound
Must happen soon after a webhook is deleted.
Errno::ENETUNREACH
Is handled the same as Errno::EHOSTUNREACH
is, in Gitlab::HTTP
.
By handling EOFError
and Errno::ENETUNREACH
they will now be logged and contribute to the failing WebHook
auto-disabling feature.
We avoid ActiveRecord::RecordNotFound
errors by having the worker return when the record is not found.
Does this MR meet the acceptance criteria?
Conformity
-
I have included changelog trailers, or none are needed. (Does this MR need a changelog?) - [-] I have added/updated documentation, or it's not needed. (Is documentation required?)
- [-] I have properly separated EE content from FOSS, or this MR is FOSS only. (Where should EE code go?)
- [-] I have added information for database reviewers in the MR description, or it's not needed. (Does this MR have database related changes?)
-
I have self-reviewed this MR per code review guidelines. -
This MR does not harm performance, or I have asked a reviewer to help assess the performance impact. (Merge request performance guidelines) -
I have followed the style guides. -
This change is backwards compatible across updates, or this does not apply.
Availability and Testing
-
I have added/updated tests following the Testing Guide, or it's not needed. (Consider all test levels. See the Test Planning Process.) - [-] I have tested this MR in all supported browsers, or it's not needed.
- [-] I have informed the Infrastructure department of a default or new setting change per definition of done, or it's not needed.
Related to #337708 (closed)