Metrics reports documentation is incomplete
Problem to solve
The documentation associated with the "metrics reports" feature introduced in !10452 (merged) is missing key information.
Further details
Specific questions and concerns:
- Which commit are the reference measurements taken from? For instance, if I open a merge request but CI has not yet finished on the current
master
commit, what will happen? - What constitutes a "change"? Will GitLab handle "noisy" measurements sensibly (e.g. look at the relative change in the metric).
- How are large numbers of metrics handled? Will the visualisation truncate the metrics list to avoid cluttering the user interface? If so, is the truncation point configurable?
- The example used in the documentation uses a textual "value"; this is in violation of the OpenMetrics specification which defines the "value" field as a floating point number.
- Apparently the metrics file needs to be both listed in
artifacts.paths
as well asartifacts.reports.metrics
in.gitlab-ci.yml
otherwise the feature silently fails, claiming that there are no changes.
Proposal
Address the questions above in the documentation.
Who can address the issue
@ayufan appears to be the author of the feature and is therefore best situated to address the issue.
Other links/references
Edited by Ben Gamari