Temporary disable `GC.compact` to avoid segfaults
What does this MR do?
After enabling nakayoshi_fork
config for Puma, which performs GC.compact
as a part of pre-forking preparation, we discovered that multiple gems, such as Hamlit
, oj
and thift
do not support GC compacting which leads to SEGFAULT in webserver processes.
An additional complication is that these failures are very random and only visible rarely, which makes them almost undetectable in CI or local envs.
To avoid the issue, our short term solution is:
- Disable
nakayoshi_fork
in production entirely: #323555 (comment 523188172) . This includes entire pre-forking routine, including 4 GC cycles and GC compacting, as they are controlled via a singlenakayoshi_fork
Puma config inpuma.rb
- Next, we want to disable the GC compacting only, but keep the 4 GC cycles preceding the compacting in Puma's
nakayoshi_fork
- After that we may want to revert https://ops.gitlab.net/gitlab-cookbooks/chef-repo/-/merge_requests/5121 and gitlab-com/gl-infra/k8s-workloads/gitlab-com!719 (merged)
- Next step is to develop a long-term solution, how and when we could advance with
GC.compact
, will be in a separate issue
Why monkey patching
This is a compromise.
To achieve a similar effect via monkey patch, we should do:
- Open 4 MRs (GitLab, GDK, CNG, Omnibus) which removes
nakayoshi_fork
line fromconfig/puma.rb
entirely - Reintroduce https://github.com/ko1/nakayoshi_fork into GitLab repo again (5th MR)
- Additionally, we will need similar 5 MRs to return back this config after finding a better solution to deal with segfaults or catch them during the CI/testing phase
Having a monkey patch, only 1 MR is needed.
Does this MR meet the acceptance criteria?
Conformity
-
📋 Does this MR need a changelog?-
I have included a changelog entry. - [-] I have not included a changelog entry because _____.
-
- [-] Documentation (if required)
-
Code review guidelines -
Merge request performance guidelines -
Style guides - [-] Database guides
- [-] Separation of EE specific content
Availability and Testing
How I tested locally:
- restarted GDK
- Noticed the message (below) in logs:
2021-03-09_09:49:13.55571 rails-web : {"timestamp":"2021-03-09T09:49:13.555Z","pid":57162,"message":"! Compacting..."}
2021-03-09_09:49:13.55575 rails-web : Note: GC compacting is currently disabled. Refer to `config/initializers_before_autoloader/003_gc_compact.rb` for details.
2021-03-09_09:49:13.55577 rails-web : {"timestamp":"2021-03-09T09:49:13.555Z","pid":57162,"message":"! Friendly fork preparation complete."}
Security
N/A
Related to #323555 (closed)
Edited by Aleksei Lipniagov