Skip to content

Adjust cron schedule for AdjournedProjectsDeletionCronWorker

What does this MR do and why?

This MR is based on the discussion at #346170 (comment 854044718), copy pasting below:

  • AdjournedGroupDeletionWorker runs at 3 am, everyday
  • AdjournedProjectsDeletionCronWorker runs at 4 am, everyday

And since there are quite a few jobs of GroupDestroyWorker enqueued from AdjournedGroupDeletionWorker, it spills over to after 4 am and at this point of time, AdjournedProjectsDeletionCronWorker is also trying to destroy some other projects.

See graph:

Screen_Shot_2022-02-24_at_2.31.06_PM

(notice that jobs overlap between 4-5 am)

Graph 1, Graph 2

Since we are running these crons on 3 am & 4 am everyday, the timing is very close to each other and we end up trying to destroy projects via both AdjournedGroupDeletionWorker & AdjournedProjectsDeletionCronWorker simultaneously from 4-5am.

If we can move AdjournedProjectsDeletionCronWorker cron to a different time everyday, say 7 am, we can comfortably avoid having too many project deletes running at the same time.

PS: There is a related follow-up in !81796 (merged)

Purpose of the change:

As stated in https://gitlab.com/gitlab-org/gitlab/-/issues/342692#note_737332055, there are a whole bunch of SQL errors/timeouts happening while removing groups (and in turn, when removing projects within those groups) via AdjournedGroupDeletionWorker. Avoiding crowding of project deletes during the same time can be the first time to make sure that we aren't stressing the database too much when these deletes happen.

Screenshots or screen recordings

These are strongly recommended to assist reviewers and reduce the time to merge your change.

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Manoj M J

Merge request reports

Loading