Upload diagnostic reports to GCS: `curl` uploader
What does this MR do and why?
This MR continues !97155 (merged), replacing no-op uploader with a real one.
Other notes
Self-managed is not affected because you need to set GITLAB_DIAGNOSTIC_REPORTS_ENABLED
. Currently, we are not working on enabling this for self-managed.
Local dev env is not affected unless you set GITLAB_DIAGNOSTIC_REPORTS_ENABLED
to start with. Also, you need FF enabled.
Obtain and configure GCS bucket for diagnostic reports
(#372239 (closed)) - I asked SRE help for that. Otherwise we'll be logging errors.
How to set up and validate locally: Uploader only
-
I tested on a real GCS bucket: and created my own GCP project via Sandbox Cloud Realm (guide)
-
Open
tail -f log/application_json.log
to check log records on success uploads or any issues -
Run
gcloud auth print-access-token
or copy your token from GCS UI. -
In
rails c
:token = "YOUR_TOKEN"; bucket = "YOUR_BUCKET_NAME"; path = "file/you/upload.json"; u = Gitlab::Memory::ReportsUploader.new(token: token, bucket: bucket); u.upload(path)
-
The file should be uploaded to the bucket, and also you will see a log record
How to set up and validate locally: Jemalloc reports + Uploader (full integration)
You will need a real GCS bucket, as described above.
We do not produce Jemalloc reports on OSX, because to pull the actual Jemalloc report, libjemalloc
must be on LD_PRELOAD
(more).
Fortunately, it is already configured in GCK this way.
So it is recommended to use GCK if you want to test it manually.
Alternatively, you could hack lib/gitlab/memory/jemalloc.rb
, so it would produce some bogus file not related to Jemalloc stats.
For the light-weigh testing (just uploader), see the section above.
First, you need to set ENV vars to enable Jemalloc reports:
GITLAB_DIAGNOSTIC_REPORTS_ENABLED: 'true'
GITLAB_DIAGNOSTIC_REPORTS_SLEEP_S: '10'
GITLAB_DIAGNOSTIC_REPORTS_SLEEP_BETWEEN_REPORTS_S: '1'
GITLAB_DIAGNOSTIC_REPORTS_SLEEP_MAX_DELTA_S: '0'
GITLAB_DIAGNOSTIC_REPORTS_PATH: '/home/git/gitlab/tmp/diag-reports'
Then, ENV vars for the uploader:
GITLAB_DIAGNOSTIC_REPORTS_UPLOADER_SLEEP_S: '10'
GITLAB_DIAGNOSTIC_REPORTS_ENABLED: 'true'
GITLAB_DIAGNOSTIC_REPORTS_GCS_TOKEN: 'YOUR-TOCKEN'
GITLAB_DIAGNOSTIC_REPORTS_GCS_BUCKET: 'YOUR-BUCKET-NAME'
You could put them into gitlab-compose-kit/docker-compose.override.yml
Second, you need to enable FF for Jemalloc: Feature.enable(:report_jemalloc_stats)
.
Then, FF for the uploader: Feature.enable(:gitlab_diagnostic_reports_uploader)
.
Restart the webserver after you set ENV vars FFs.
The reports will be produced and uploaded, you should also notice that the /tmp
subdir under the configured GITLAB_DIAGNOSTIC_REPORTS_PATH
should appear.
Check GCS bucket for the uploads.
Check tail -f log/application_json.log
in case of any errors.
In my GCK, I was hitting "curl: option --fail-with-body: is unknown\ncurl: try 'curl --help' or 'curl --manual' for more information\n"
, although I expect us to have the latest curl in prod: https://gitlab.com/gitlab-org/build/CNG/-/blob/master/gitlab-base/Dockerfile#L14.
So I needed to remove this line temporarily for local env.
My local GCK uses 7.74
, the option was introduced in 7.76
. Probably rebuilding GCK will help?
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #372242 (closed)