Use Mutex and limit threads by pool size in GPT Generator
Update Generator to use Mutex and sync thread counts with pool size. With this update Generator will create GPT_GENERATOR_POOL_SIZE
threads and each of them will send multiple requests. Previous configuration was not efficient as it tried to create threads for each project/group.
Changes:
- Fixes active connection limitation issue for projects - #486 (closed)
- Fixes Generator Group count issue if pool size wasn't divided correctly when a custom pool size was used - #485 (closed)
- Bumps Generator version to
1.0.23
As can be seen from the charts below, active connection now increases correctly (grafana):
Before update - 2 minutes 57 seconds with incorrect 40 groups instead of 50:
Local test - 50 groups x 10 projects with pool size 20
GPT Data Generator v1.0.22 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://localhost' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://localhost, Version: 14.6.0-pre 49ec55172c1
Creating group gpt_5_old
Creating group gpt_5_old/many_groups_and_projects
Creating 50 groups with name prefix 'gpt-subgroup-' under parent group 'gpt_5_old/many_groups_and_projects'
........................................
Checking for existing projects under groups...
........................................
Creating 10 projects each under 40 subgroups with name prefix 'gpt-project-'
................................................................................................................................................................................................................................................................................................................................................................................................................
<-> Horizontal data: successfully generated after 2 minutes 57 seconds!
█ GPT data generation finished after 2 minutes 57 seconds.
After update - 2 minutes 30 seconds with correct 50 groups:
Local test - 50 groups x 10 projects with pool size 20 -
GPT Data Generator v1.0.23 - opinionated test data for the GitLab Performance Tool
Checking that GitLab environment 'http://localhost' is available, supported and that provided Access Token works...
Environment and Access Token check complete - URL: http://localhost, Version: 14.6.0-pre 49ec55172c1
Creating group gpt_5_new
Creating group gpt_5_new/many_groups_and_projects
Creating 50 groups with name prefix 'gpt-subgroup-' under parent group 'gpt_5_new/many_groups_and_projects'
..................................................
Checking for existing projects under groups...
..................................................
Creating 10 projects each under 50 subgroups with name prefix 'gpt-project-'
....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
<-> Horizontal data: successfully generated after 2 minutes 30 seconds!
█ GPT data generation finished after 2 minutes 30 seconds.
█ Logs: results/generate-gpt-data_localhost_2021-12-03_231315.log
Closes #486 (closed) #485 (closed)