ci: faster test coverage
We can run unit tests with the -coverprofile
flag and use go tool cover
to analyse the results to get a report on the code coverage.
Before this change, the coverage profile was generated per package, and then merged into a single coverage.txt
file. However, this approach doesn't work with the go tool cover
because of some extra lines that exist in the merged file.
With this MR, we can run all unit tests defined in $COVERAGE_PACKAGES
(excludes integration tests) and generate the report from a single file. We can then call go tool cover
on the result, and use to report coverage
in the CI yaml file.
Results comparison
Time:
time make short-cover
...
make short-cover 22.78s user 5.77s system 124% cpu 22.856 total
vs
time make coverage
...
make coverage 36.30s user 33.71s system 50% cpu 2:19.44 total
Both commands output two files coverage.txt
and short-coverage.txt
, we can then compare by first doing:
- remove all lines that contain the word
atomic
- sort both files
- diff sorted-coverage.out sorted-short-coverage.out
Click to expand
diff sorted-cov.out sorted-short-cov.out
992c992
< github.com/docker/distribution/notifications/sinks.go:197.26,198.16 1 3
---
> github.com/docker/distribution/notifications/sinks.go:197.26,198.16 1 2
994c994
< github.com/docker/distribution/notifications/sinks.go:203.3,203.17 1 2
---
> github.com/docker/distribution/notifications/sinks.go:203.3,203.17 1 1
1013c1013
< github.com/docker/distribution/notifications/sinks.go:307.2,307.15 1 333
---
> github.com/docker/distribution/notifications/sinks.go:307.2,307.15 1 459
1015,1018c1015,1018
< github.com/docker/distribution/notifications/sinks.go:311.19,314.13 3 244
< github.com/docker/distribution/notifications/sinks.go:311.2,311.19 1 332
< github.com/docker/distribution/notifications/sinks.go:317.2,317.40 1 88
< github.com/docker/distribution/notifications/sinks.go:317.40,318.27 1 78
---
> github.com/docker/distribution/notifications/sinks.go:311.19,314.13 3 339
> github.com/docker/distribution/notifications/sinks.go:311.2,311.19 1 458
> github.com/docker/distribution/notifications/sinks.go:317.2,317.40 1 119
> github.com/docker/distribution/notifications/sinks.go:317.40,318.27 1 109
1020c1020
< github.com/docker/distribution/notifications/sinks.go:323.3,324.13 2 78
---
> github.com/docker/distribution/notifications/sinks.go:323.3,324.13 2 109
1025,1026c1025,1026
< github.com/docker/distribution/notifications/sinks.go:345.51,346.45 1 88
< github.com/docker/distribution/notifications/sinks.go:346.45,349.3 2 78
---
> github.com/docker/distribution/notifications/sinks.go:345.51,346.45 1 119
> github.com/docker/distribution/notifications/sinks.go:346.45,349.3 2 109
1028c1028
< github.com/docker/distribution/notifications/sinks.go:357.53,363.2 3 244
---
> github.com/docker/distribution/notifications/sinks.go:357.53,363.2 3 339
1030,1031c1030,1031
< github.com/docker/distribution/notifications/sinks.go:372.35,375.2 2 78
< github.com/docker/distribution/notifications/sinks.go:379.40,382.2 1 332
---
> github.com/docker/distribution/notifications/sinks.go:372.35,375.2 2 109
> github.com/docker/distribution/notifications/sinks.go:379.40,382.2 1 458
5168c5168
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:123.12,125.3 1 22
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:123.12,125.3 1 27
5170c5170
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:127.2,127.71 1 208
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:127.2,127.71 1 203
5172,5173c5172,5173
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:131.2,133.8 3 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:133.8,137.3 3 201
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:131.2,133.8 3 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:133.8,137.3 3 196
5205,5206c5205,5206
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:37.102,42.2 4 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:44.132,46.16 2 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:37.102,42.2 4 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:44.132,46.16 2 210
5208,5210c5208,5210
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.2,50.47 1 215
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.47,52.3 1 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:54.2,55.16 2 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.2,50.47 1 210
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.47,52.3 1 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:54.2,55.16 2 210
5212c5212
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:59.2,62.16 3 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:59.2,62.16 3 210
5214c5214
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:66.2,68.18 2 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:66.2,68.18 2 210
5216,5218c5216,5218
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:73.16,77.3 1 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.16,82.3 2 22
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.2,79.16 1 22
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:73.16,77.3 1 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.16,82.3 2 27
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.2,79.16 1 27
5776c5776
< github.com/docker/distribution/registry/storage/blobstore.go:101.23,103.4 1 461
---
> github.com/docker/distribution/registry/storage/blobstore.go:101.23,103.4 1 463
5827c5827
< github.com/docker/distribution/registry/storage/blobstore.go:99.84,101.23 1 661
---
> github.com/docker/distribution/registry/storage/blobstore.go:99.84,101.23 1 663
6076,6083c6076,6083
< github.com/docker/distribution/registry/storage/catalog.go:101.17,103.3 1 11
< github.com/docker/distribution/registry/storage/catalog.go:105.2,105.25 1 73
< github.com/docker/distribution/registry/storage/catalog.go:105.25,107.16 2 116
< github.com/docker/distribution/registry/storage/catalog.go:107.16,109.4 1 17
< github.com/docker/distribution/registry/storage/catalog.go:111.16,113.4 1 15
< github.com/docker/distribution/registry/storage/catalog.go:111.3,111.16 1 116
< github.com/docker/distribution/registry/storage/catalog.go:115.14,117.4 1 17
< github.com/docker/distribution/registry/storage/catalog.go:115.3,115.14 1 116
---
> github.com/docker/distribution/registry/storage/catalog.go:101.17,103.3 1 13
> github.com/docker/distribution/registry/storage/catalog.go:105.2,105.25 1 71
> github.com/docker/distribution/registry/storage/catalog.go:105.25,107.16 2 114
> github.com/docker/distribution/registry/storage/catalog.go:107.16,109.4 1 16
> github.com/docker/distribution/registry/storage/catalog.go:111.16,113.4 1 16
> github.com/docker/distribution/registry/storage/catalog.go:111.3,111.16 1 114
> github.com/docker/distribution/registry/storage/catalog.go:115.14,117.4 1 15
> github.com/docker/distribution/registry/storage/catalog.go:115.3,115.14 1 114
6124,6125c6124,6125
< github.com/docker/distribution/registry/storage/catalog.go:88.33,91.2 1 73
< github.com/docker/distribution/registry/storage/catalog.go:95.61,101.17 2 73
---
> github.com/docker/distribution/registry/storage/catalog.go:88.33,91.2 1 71
> github.com/docker/distribution/registry/storage/catalog.go:95.61,101.17 2 71
6184c6184
< github.com/docker/distribution/registry/storage/filereader.go:110.51,111.19 1 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:110.51,111.19 1 2812
6186,6188c6186,6188
< github.com/docker/distribution/registry/storage/filereader.go:115.18,117.3 1 2188
< github.com/docker/distribution/registry/storage/filereader.go:115.2,115.18 1 3218
< github.com/docker/distribution/registry/storage/filereader.go:120.2,121.16 2 1030
---
> github.com/docker/distribution/registry/storage/filereader.go:115.18,117.3 1 1781
> github.com/docker/distribution/registry/storage/filereader.go:115.2,115.18 1 2812
> github.com/docker/distribution/registry/storage/filereader.go:120.2,121.16 2 1031
6192c6192
< github.com/docker/distribution/registry/storage/filereader.go:134.2,136.19 2 1029
---
> github.com/docker/distribution/registry/storage/filereader.go:134.2,136.19 2 1030
6194,6196c6194,6196
< github.com/docker/distribution/registry/storage/filereader.go:138.8,140.3 1 1023
< github.com/docker/distribution/registry/storage/filereader.go:142.2,142.20 1 1029
< github.com/docker/distribution/registry/storage/filereader.go:149.31,150.19 1 1026
---
> github.com/docker/distribution/registry/storage/filereader.go:138.8,140.3 1 1024
> github.com/docker/distribution/registry/storage/filereader.go:142.2,142.20 1 1030
> github.com/docker/distribution/registry/storage/filereader.go:149.31,150.19 1 1027
6198,6199c6198,6199
< github.com/docker/distribution/registry/storage/filereader.go:153.18,156.3 2 1023
< github.com/docker/distribution/registry/storage/filereader.go:153.2,153.18 1 1026
---
> github.com/docker/distribution/registry/storage/filereader.go:153.18,156.3 2 1024
> github.com/docker/distribution/registry/storage/filereader.go:153.2,153.18 1 1027
6206c6206
< github.com/docker/distribution/registry/storage/filereader.go:52.57,53.19 1 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:52.57,53.19 1 2812
6208c6208
< github.com/docker/distribution/registry/storage/filereader.go:57.2,58.16 2 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:57.2,58.16 2 2812
6210c6210
< github.com/docker/distribution/registry/storage/filereader.go:62.2,66.40 3 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:62.2,66.40 3 2812
6212c6212
< github.com/docker/distribution/registry/storage/filereader.go:70.2,70.15 1 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:70.2,70.15 1 2812
6222c6222
< github.com/docker/distribution/registry/storage/filereader.go:93.29,95.4 1 1026
---
> github.com/docker/distribution/registry/storage/filereader.go:93.29,95.4 1 1027
Check this URL for a simpler diff.
There are only 30 diffs, out of 7011 lines. Checking them in a bit more detail, it seems like a small difference in the way the coverage tool generates some of the numbers for comparison. Based on this, I think the diff is quite insignificant so we can opt for the faster, shorter way of running the test coverage.