Draft: Benchmark CI Header validation
What does this MR do and why?
This MR benchmarks the validation speed of the CI Header when using Ruby classes vs using a JSON schema.
Issue: #467375 (closed)
Benchmarking (times in milliseconds)
1. JSON schema is loaded for every run - benchmark_json_schema_no_preload.rb
-
JsonSchemaValidator
loads the schema for every validation it performs. This benchmark tests that method - Ruby validation is faster in the majority of runs
Validation method | Number of runs | user (ms) | system (ms) | total (ms) | real (ms) |
---|---|---|---|---|---|
Ruby | 1 | 0.667 | 0.067 | 0.734 | (0.727) |
JSON schema | 1 | 0.400 | 0.158 | 0.558 | (0.559) |
Ruby | 5 | 0.774 | 0.010 | 0.784 | (0.784) |
JSON schema | 5 | 0.903 | 0.170 | 1.073 | (1.074) |
Ruby | 50 | 7.195 | 0.067 | 7.262 | (7.262) |
JSON schema | 50 | 8.398 | 1.799 | 10.197 | (10.199) |
Ruby | 500 | 72.926 | 1.159 | 74.085 | (74.085) |
JSON schema | 500 | 79.843 | 17.520 | 97.363 | (97.371) |
2. JSON schema is preloaded and reused between runs - benchmark_json_schema_preload.rb
- By preloading the JSON schema, we greatly reduce the JSON schema validation time
- JSON schema validation is now consistently 3 - 4x faster than Ruby validation
Validation method | Number of runs | user (ms) | system (ms) | total (ms) | real (ms) |
---|---|---|---|---|---|
Ruby | 1 | 0.174 | 0.003 | 0.177 | (0.176) |
JSON schema | 1 | 0.060 | 0.002 | 0.062 | (0.061) |
Ruby | 5 | 0.866 | 0.068 | 0.934 | (0.934) |
JSON schema | 5 | 0.236 | 0.008 | 0.244 | (0.245) |
Ruby | 50 | 7.941 | 0.601 | 8.542 | (8.544) |
JSON schema | 50 | 2.379 | 0.104 | 2.483 | (2.487) |
Ruby | 500 | 72.788 | 1.734 | 74.522 | (74.525) |
JSON schema | 500 | 23.779 | 0.646 | 24.425 | (24.429) |
Edited by Avielle Wolfe