Add script to migrate InfluxDB e2e test data to GCS in json format
What does this MR do and why?
Closes gitlab-org/quality/quality-engineering/team-tasks#2703 (closed)
This MR:
- Converts
MigrateInfluxDataToGcs
to a super class and moves CSV specific functionality toMigrateInfluxDataToGcsCsv
- Adds
MigrateInfluxDataToGcsJson
that creates json objects and uploads to a GCP bucket.
Please note that there are no code changes in MigrateInfluxDataToGcsCsv
. It has the same code moved over from MigrateInfluxDataToGcs
.
- An example run of
bundle exec rake "influx_to_gcs_json[2024, 1]"
: https://gitlab.com/gitlab-org/quality/toolbox/-/jobs/6902606810 - The bucket where the files were uploaded: https://console.cloud.google.com/storage/browser/e2e_test_metrics_json;tab=objects?forceOnBucketsSortingFiltering=true&authuser=1&project=gitlab-qa-resources&prefix=&forceOnObjectsSortingFiltering=false
MR acceptance checklist
Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Screenshots or screen recordings
Screenshots are required for UI changes, and strongly recommended for all other merge requests.
Before | After |
---|---|
How to set up and validate locally
-
Download and save the json file in 1P attached to a note titled "influx-to-gcs GCS service account in gitlab-qa-resources". The path to the saved file should be provided in
QA_METRICS_GCS_CREDS
in the below command. -
From the
qa
folder, run:
QA_INFLUXDB_TIMEOUT=300 QA_METRICS_GCS_PROJECT_ID="gitlab-qa-resources" QA_METRICS_GCS_CREDS="/path/to/json/key/file.json" QA_METRICS_GCS_BUCKET_NAME="e2e_test_metrics_json" QA_INFLUXDB_TOKEN="<QA_INFLUXDB_TOKEN_FROM_1P>" QA_INFLUXDB_URL="https://influxdb.quality.gitlab.net" bundle exec rake "influx_to_gcs_json[2024, 1, 1]"
Edited by Sanad Liaquat