Rescue other errors when checking redis' connectivity
Context
When checking Redis connectivity in https://gitlab.com/gitlab-org/build/CNG/blob/master/gitlab-rails/scripts/lib/checks/redis.rb, the following exceptions are sometimes raised:
-
Errno::EALREADY: Operation already in progress - connect(2) for 10.0.13.23:6379
(see GCP logs) -
__connect_nonblock': Operation now in progress - connect(2) would block (IO::EINPROGRESSWaitWritable)
(see GCP logs)
Those exceptions are not being correctly caught at the moment. This is because we only catch RuntimeError
exceptions, and Errno::*
errors and IO::EINPROGRESSWaitWritable
are not children errors of RuntimeError
(source). This results in a forceful exit of the Checks::Redis
class.
What does this MR do?
We also catch SystemCallError
exceptions, because it will catch both of the exceptions mentioned above:
irb> Errno::EALREADY < SystemCallError
true
irb> IO::EINPROGRESSWaitWritable < SystemCallError
true
Related issues
gitlab-org/quality/engineering-productivity/review-apps-broken-incidents#30 (closed)
Checklist
See Definition of done.
For anything in this list which will not be completed, please provide a reason in the MR discussion
Required
-
Merge Request Title, and Description are up to date, accurate, and descriptive -
MR targeting the appropriate branch -
MR has a green pipeline on GitLab.com
Expected (please provide an explanation if not completing)
-
Test plan indicating conditions for success has been posted and passes -
Documentation created/updated -
Integration tests added to GitLab QA -
The impact any change in container size has should be evaluated