Skip to content

Push timing & GC measurements down to the service layer

The following discussion from !28445 (merged) should be addressed:

Several times now the question came up how to log GC & timing measurements emitted by the measurable module (#209781 (closed), #208930). It was suggested to use a structured JSON log instead of logging plain text to stdout. Before we do that, we should make sure that the measurements can actually be taken outside of a simple rake task execution, since that's the only place where we currently log these.

If we would push the measurements down from the rake layer to the service layer (where the actual import/export logic executes), then we would be able to hook this up to imports/exports triggered from API calls as well. This in turns means we'd be able to take measurements via our performance CI pipeline, which is currently based on API integration, not rake executions. Only then it would make sense to consider structured logging so that these logs could be indexed by Kibana for instance.

Summary:

  • call with_measuring not in the rake tasks, but in the import/export services
  • make sure that measurement is behind the feature flag
  • rake tasks should output measurement to STDOUT if DEBUG mode is enabled
  • log measurements in structured (JSON) format in the new log file service_measurement.log.
  • remove measurement options from a rake task

Feature Flags:

We will enable measuring for:

  • Projects::ImportExport::ExportService - it will be behind the feature flag measure_project_export_service
  • Projects::ImportService - it will be behind the feature flag measure_project_import_service
  • Projects::CreateService - it will be behind the feature flag measure_project_create_service

Additional steps when introducing new log file

  1. Consider log retention settings. By default, Omnibus will rotate any logs in /var/log/gitlab/gitlab-rails/*.log every hour and keep at most 30 compressed files. On GitLab.com, that setting is only 6 compressed files. These settings should suffice for most users, but you may need to tweak them in Omnibus GitLab.

  2. If you add a new file, submit an issue to the production tracker or a merge request to the gitlab_fluentd project. See this example.

  3. Be sure to update the GitLab CE/EE documentation and the GitLab.com runbooks.

Edited by Nikola Milojevic