Instrument telemetry middleware
Part of https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/23887
Example log entries:
{"accepted_request_count": 1, "total_request_count": 2, "error_request_count": 3, "model_engine": "vertex-ai", "model_name": "text-bison@001", "correlation_id": "fa780c2ca5104cf691843cc504d7121e", "logger": "telemetry", "level": "info", "type": "mlops", "stage": "main", "timestamp": "2023-06-09T21:45:30.699321Z", "message": "telemetry"}
{"model_engine": "vertex-ai", "model_name": "text-bison@001", "correlation_id": "6aa577d3c6a64b8cab501c0eee9f8112", "logger": "telemetry", "level": "error", "type": "mlops", "stage": "main", "timestamp": "2023-06-09T21:51:50.337644Z", "message": "failed to capture model telemetry: 1 validation error for ModelTelemetry\naccepted_request_count\n value is not a valid integer (type=type_error.integer)"}
Example metrics:
# HELP code_suggestions_accepts_total Accepts count by number
# TYPE code_suggestions_accepts_total counter
code_suggestions_accepts_total{model_engine="vertex-ai",model_name="text-bison@001"} 1.0
# HELP code_suggestions_accepts_created Accepts count by number
# TYPE code_suggestions_accepts_created gauge
code_suggestions_accepts_created{model_engine="vertex-ai",model_name="text-bison@001"} 1.686347130700104e+09
# HELP code_suggestions_requests_total Requests count by number
# TYPE code_suggestions_requests_total counter
code_suggestions_requests_total{model_engine="vertex-ai",model_name="text-bison@001"} 2.0
# HELP code_suggestions_requests_created Requests count by number
# TYPE code_suggestions_requests_created gauge
code_suggestions_requests_created{model_engine="vertex-ai",model_name="text-bison@001"} 1.686347130700143e+09
# HELP code_suggestions_errors_total Errors count by number
# TYPE code_suggestions_errors_total counter
code_suggestions_errors_total{model_engine="vertex-ai",model_name="text-bison@001"} 3.0
# HELP code_suggestions_errors_created Errors count by number
# TYPE code_suggestions_errors_created gauge
code_suggestions_errors_created{model_engine="vertex-ai",model_name="text-bison@001"} 1.686347130700172e+09
Edited by Alejandro Rodríguez