supervisor: Fix bugs related to timeouts
This MR fixes two bugs related to timeouts in the supervisor package I found while investigating #3872 (closed). I didn't find the root cause for the flakiness unfortunately, but fixing real bugs doesn't hurt either.