Skip to content

Correct calculations for vertical and horizontal gpt data generation

John McDonnell requested to merge jmd/fix-run-duration-logs into main

Description

#561 (comment 1384972505) noted an error in how we report the vertical data duration

NB -- should the log for duration of Vertical data: in the above read 1:20:56? It looks to actually report Horizontal+Vertical duration

Rectify this by adding a timer specific to vertical and horizontal data creation.

Sample Result

| Horizontal data: successfully generated after 10 seconds!
| Vertical data: successfully generated after 5 seconds!
█ GPT data generation finished after 18 seconds.
Expand to see sample output 
# mocked out `create_vertical_test_data` / `create_horizontal_test_data` with calls to sleep 
$ yes | bundle exec bin/generate-gpt-data --environment localhost.json --vertical --horizontal

GPT Data Generator v1.2.1 - opinionated test data for the GitLab Performance Tool
The GPT Data Generator will inject the data into the specified group `gpt` with Public visibility on http://gdk.test:3000. Note that this may take some time.
Do you want to proceed? [Y/N]
Checking that GitLab environment 'http://gdk.test:3000' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://gdk.test:3000, Version: 16.4.0-pre 0daa71593c6 {"enabled"=>false, "externalUrl"=>nil, "version"=>nil} true

For the creation of `horizontal` data with large projects, the GPT Data Generator will need to change the Repository Storages setting on the target GitLab environment. This is to facilitate the creation of numerous Groups and Projects evenly across each Storage node. As such, it will change this setting to point to all nodes as set in the `storage_nodes` option and then create the data.

While the Generator is doing this any other projects created or imported during this time will be stored on one of these nodes randomly.
The original setting will be restored after the the tool is finished.
Do you want to proceed? [Y/N]
Group gpt already exists
Group gpt/many_groups_and_projects already exists

| Horizontal data: successfully generated after 10 seconds!

For the creation of `vertical` data with large projects, the GPT Data Generator will need to change the Repository Storages setting on the target GitLab environment. This is to facilitate the creation of a large project in each Storage node specifically. As such, it will change this setting to point to each node as set in the `storage_nodes` option and then create the project sequentially.

While the Generator is doing this any other projects created or imported during this time will also be confined to the currently active Storage node.
The original setting will be restored after the the script is finished.
Do you want to proceed? [Y/N]

| Vertical data: importing large projects for GPT...
Group gpt already exists
Group gpt/large_projects already exists

| Vertical data: successfully generated after 5 seconds!
█ GPT data generation finished after 18 seconds.

█ Logs: results/generate-gpt-data_gdk.test_2023-09-05_122748.log

Merge request reports