Make constant-memory export serializer
What does this MR do?
Make constant-memory export serializer
This makes the serialization to be executed lazily, using memory-optimised approach, and being fast to recycle.
The biggest benefit comes from the fact that storing full Hash of object serialization is expensive, as it also often results in holding original representations of objects.
This solves that, by making batch serialization, that outputs exact raw JSON content to generated string.
JSON generator is optimised to have efficient string appender, this makes use of that.
This brings big memory usage benefits as described in: https://gitlab.com/gitlab-org/gitlab-ce/issues/35389#note_211182737 and confirmed by two independent installations.
- Shorten export by about 4x times with N+1 fix,
- We can reduce memory footprint (for gitlabhq) from around 700MB (on top of process), to around 150MB.
Does this MR meet the acceptance criteria?
Conformity
- [-] Changelog entry
- [-] Documentation created/updated or follow-up review issue created
-
Code review guidelines -
Merge request performance guidelines -
Style guides - [-] Database guides
- [-] [Separation of EE specific content](https://docs.gitlab.com/ee/development/ee_features.html#separation-of-ee-code
Related to #26910 (closed)
Edited by 🤖 GitLab Bot 🤖