Skip to content

Draft: Benchmark CI Header validation

What does this MR do and why?

This MR benchmarks the validation speed of the CI Header when using Ruby classes vs using a JSON schema.

Issue: #467375 (closed)

Benchmarking (times in milliseconds)

1. JSON schema is loaded for every run - benchmark_json_schema_no_preload.rb

  • JsonSchemaValidator loads the schema for every validation it performs. This benchmark tests that method
  • Ruby validation is faster in the majority of runs
Validation method Number of runs user (ms) system (ms) total (ms) real (ms)
Ruby 1 0.667     0.067 0.734 (0.727)
JSON schema 1 0.400 0.158 0.558 (0.559)
Ruby 5 0.774 0.010 0.784 (0.784)
JSON schema 5 0.903 0.170 1.073 (1.074)
Ruby 50 7.195 0.067 7.262 (7.262)
JSON schema 50 8.398 1.799 10.197 (10.199)
Ruby 500 72.926 1.159 74.085 (74.085)
JSON schema 500 79.843 17.520 97.363 (97.371)

2. JSON schema is preloaded and reused between runs - benchmark_json_schema_preload.rb

  • By preloading the JSON schema, we greatly reduce the JSON schema validation time
  • JSON schema validation is now consistently 3 - 4x faster than Ruby validation
Validation method Number of runs user (ms) system (ms) total (ms) real (ms)
Ruby 1 0.174    0.003 0.177 (0.176)
JSON schema 1 0.060 0.002 0.062 (0.061)
Ruby 5 0.866 0.068 0.934 (0.934)
JSON schema 5 0.236 0.008 0.244 (0.245)
Ruby 50 7.941 0.601 8.542 (8.544)
JSON schema 50 2.379 0.104 2.483 (2.487)
Ruby 500 72.788 1.734 74.522 (74.525)
JSON schema 500 23.779 0.646 24.425 (24.429)
Edited by Avielle Wolfe

Merge request reports

Loading