Adjust cron schedule for AdjournedProjectsDeletionCronWorker
What does this MR do and why?
This MR is based on the discussion at #346170 (comment 854044718), copy pasting below:
-
AdjournedGroupDeletionWorker
runs at 3 am, everyday -
AdjournedProjectsDeletionCronWorker
runs at 4 am, everyday
And since there are quite a few jobs of GroupDestroyWorker
enqueued from AdjournedGroupDeletionWorker
, it spills over to after 4 am and at this point of time, AdjournedProjectsDeletionCronWorker
is also trying to destroy some other projects.
See graph:
(notice that jobs overlap between 4-5 am)
Since we are running these crons on 3 am & 4 am everyday, the timing is very close to each other and we end up trying to destroy projects via both AdjournedGroupDeletionWorker
& AdjournedProjectsDeletionCronWorker
simultaneously from 4-5am.
If we can move AdjournedProjectsDeletionCronWorker
cron to a different time everyday, say 7 am, we can comfortably avoid having too many project deletes running at the same time.
PS: There is a related follow-up in !81796 (merged)
Purpose of the change:
As stated in https://gitlab.com/gitlab-org/gitlab/-/issues/342692#note_737332055, there are a whole bunch of SQL errors/timeouts happening while removing groups (and in turn, when removing projects within those groups) via AdjournedGroupDeletionWorker
. Avoiding crowding of project deletes during the same time can be the first time to make sure that we aren't stressing the database too much when these deletes happen.
Screenshots or screen recordings
These are strongly recommended to assist reviewers and reduce the time to merge your change.
How to set up and validate locally
Numbered steps to set up and validate the change are strongly suggested.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.