ci: faster test coverage

We can run unit tests with the -coverprofile flag and use go tool cover to analyse the results to get a report on the code coverage. Before this change, the coverage profile was generated per package, and then merged into a single coverage.txt file. However, this approach doesn't work with the go tool cover because of some extra lines that exist in the merged file.

With this MR, we can run all unit tests defined in $COVERAGE_PACKAGES (excludes integration tests) and generate the report from a single file. We can then call go tool cover on the result, and use to report coverage in the CI yaml file.

Results comparison

Time:

time make short-cover
...
make short-cover  22.78s user 5.77s system 124% cpu 22.856 total

vs

time make coverage
...
make coverage  36.30s user 33.71s system 50% cpu 2:19.44 total

Both commands output two files coverage.txt and short-coverage.txt, we can then compare by first doing:

1. remove all lines that contain the word atomic
2. sort both files
3. diff sorted-coverage.out sorted-short-coverage.out

Click to expand

diff sorted-cov.out sorted-short-cov.out
992c992
< github.com/docker/distribution/notifications/sinks.go:197.26,198.16 1 3
---
> github.com/docker/distribution/notifications/sinks.go:197.26,198.16 1 2
994c994
< github.com/docker/distribution/notifications/sinks.go:203.3,203.17 1 2
---
> github.com/docker/distribution/notifications/sinks.go:203.3,203.17 1 1
1013c1013
< github.com/docker/distribution/notifications/sinks.go:307.2,307.15 1 333
---
> github.com/docker/distribution/notifications/sinks.go:307.2,307.15 1 459
1015,1018c1015,1018
< github.com/docker/distribution/notifications/sinks.go:311.19,314.13 3 244
< github.com/docker/distribution/notifications/sinks.go:311.2,311.19 1 332
< github.com/docker/distribution/notifications/sinks.go:317.2,317.40 1 88
< github.com/docker/distribution/notifications/sinks.go:317.40,318.27 1 78
---
> github.com/docker/distribution/notifications/sinks.go:311.19,314.13 3 339
> github.com/docker/distribution/notifications/sinks.go:311.2,311.19 1 458
> github.com/docker/distribution/notifications/sinks.go:317.2,317.40 1 119
> github.com/docker/distribution/notifications/sinks.go:317.40,318.27 1 109
1020c1020
< github.com/docker/distribution/notifications/sinks.go:323.3,324.13 2 78
---
> github.com/docker/distribution/notifications/sinks.go:323.3,324.13 2 109
1025,1026c1025,1026
< github.com/docker/distribution/notifications/sinks.go:345.51,346.45 1 88
< github.com/docker/distribution/notifications/sinks.go:346.45,349.3 2 78
---
> github.com/docker/distribution/notifications/sinks.go:345.51,346.45 1 119
> github.com/docker/distribution/notifications/sinks.go:346.45,349.3 2 109
1028c1028
< github.com/docker/distribution/notifications/sinks.go:357.53,363.2 3 244
---
> github.com/docker/distribution/notifications/sinks.go:357.53,363.2 3 339
1030,1031c1030,1031
< github.com/docker/distribution/notifications/sinks.go:372.35,375.2 2 78
< github.com/docker/distribution/notifications/sinks.go:379.40,382.2 1 332
---
> github.com/docker/distribution/notifications/sinks.go:372.35,375.2 2 109
> github.com/docker/distribution/notifications/sinks.go:379.40,382.2 1 458
5168c5168
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:123.12,125.3 1 22
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:123.12,125.3 1 27
5170c5170
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:127.2,127.71 1 208
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:127.2,127.71 1 203
5172,5173c5172,5173
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:131.2,133.8 3 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:133.8,137.3 3 201
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:131.2,133.8 3 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:133.8,137.3 3 196
5205,5206c5205,5206
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:37.102,42.2 4 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:44.132,46.16 2 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:37.102,42.2 4 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:44.132,46.16 2 210
5208,5210c5208,5210
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.2,50.47 1 215
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.47,52.3 1 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:54.2,55.16 2 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.2,50.47 1 210
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:50.47,52.3 1 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:54.2,55.16 2 210
5212c5212
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:59.2,62.16 3 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:59.2,62.16 3 210
5214c5214
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:66.2,68.18 2 215
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:66.2,68.18 2 210
5216,5218c5216,5218
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:73.16,77.3 1 208
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.16,82.3 2 22
< github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.2,79.16 1 22
---
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:73.16,77.3 1 203
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.16,82.3 2 27
> github.com/docker/distribution/registry/proxy/proxyblobstore.go:79.2,79.16 1 27
5776c5776
< github.com/docker/distribution/registry/storage/blobstore.go:101.23,103.4 1 461
---
> github.com/docker/distribution/registry/storage/blobstore.go:101.23,103.4 1 463
5827c5827
< github.com/docker/distribution/registry/storage/blobstore.go:99.84,101.23 1 661
---
> github.com/docker/distribution/registry/storage/blobstore.go:99.84,101.23 1 663
6076,6083c6076,6083
< github.com/docker/distribution/registry/storage/catalog.go:101.17,103.3 1 11
< github.com/docker/distribution/registry/storage/catalog.go:105.2,105.25 1 73
< github.com/docker/distribution/registry/storage/catalog.go:105.25,107.16 2 116
< github.com/docker/distribution/registry/storage/catalog.go:107.16,109.4 1 17
< github.com/docker/distribution/registry/storage/catalog.go:111.16,113.4 1 15
< github.com/docker/distribution/registry/storage/catalog.go:111.3,111.16 1 116
< github.com/docker/distribution/registry/storage/catalog.go:115.14,117.4 1 17
< github.com/docker/distribution/registry/storage/catalog.go:115.3,115.14 1 116
---
> github.com/docker/distribution/registry/storage/catalog.go:101.17,103.3 1 13
> github.com/docker/distribution/registry/storage/catalog.go:105.2,105.25 1 71
> github.com/docker/distribution/registry/storage/catalog.go:105.25,107.16 2 114
> github.com/docker/distribution/registry/storage/catalog.go:107.16,109.4 1 16
> github.com/docker/distribution/registry/storage/catalog.go:111.16,113.4 1 16
> github.com/docker/distribution/registry/storage/catalog.go:111.3,111.16 1 114
> github.com/docker/distribution/registry/storage/catalog.go:115.14,117.4 1 15
> github.com/docker/distribution/registry/storage/catalog.go:115.3,115.14 1 114
6124,6125c6124,6125
< github.com/docker/distribution/registry/storage/catalog.go:88.33,91.2 1 73
< github.com/docker/distribution/registry/storage/catalog.go:95.61,101.17 2 73
---
> github.com/docker/distribution/registry/storage/catalog.go:88.33,91.2 1 71
> github.com/docker/distribution/registry/storage/catalog.go:95.61,101.17 2 71
6184c6184
< github.com/docker/distribution/registry/storage/filereader.go:110.51,111.19 1 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:110.51,111.19 1 2812
6186,6188c6186,6188
< github.com/docker/distribution/registry/storage/filereader.go:115.18,117.3 1 2188
< github.com/docker/distribution/registry/storage/filereader.go:115.2,115.18 1 3218
< github.com/docker/distribution/registry/storage/filereader.go:120.2,121.16 2 1030
---
> github.com/docker/distribution/registry/storage/filereader.go:115.18,117.3 1 1781
> github.com/docker/distribution/registry/storage/filereader.go:115.2,115.18 1 2812
> github.com/docker/distribution/registry/storage/filereader.go:120.2,121.16 2 1031
6192c6192
< github.com/docker/distribution/registry/storage/filereader.go:134.2,136.19 2 1029
---
> github.com/docker/distribution/registry/storage/filereader.go:134.2,136.19 2 1030
6194,6196c6194,6196
< github.com/docker/distribution/registry/storage/filereader.go:138.8,140.3 1 1023
< github.com/docker/distribution/registry/storage/filereader.go:142.2,142.20 1 1029
< github.com/docker/distribution/registry/storage/filereader.go:149.31,150.19 1 1026
---
> github.com/docker/distribution/registry/storage/filereader.go:138.8,140.3 1 1024
> github.com/docker/distribution/registry/storage/filereader.go:142.2,142.20 1 1030
> github.com/docker/distribution/registry/storage/filereader.go:149.31,150.19 1 1027
6198,6199c6198,6199
< github.com/docker/distribution/registry/storage/filereader.go:153.18,156.3 2 1023
< github.com/docker/distribution/registry/storage/filereader.go:153.2,153.18 1 1026
---
> github.com/docker/distribution/registry/storage/filereader.go:153.18,156.3 2 1024
> github.com/docker/distribution/registry/storage/filereader.go:153.2,153.18 1 1027
6206c6206
< github.com/docker/distribution/registry/storage/filereader.go:52.57,53.19 1 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:52.57,53.19 1 2812
6208c6208
< github.com/docker/distribution/registry/storage/filereader.go:57.2,58.16 2 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:57.2,58.16 2 2812
6210c6210
< github.com/docker/distribution/registry/storage/filereader.go:62.2,66.40 3 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:62.2,66.40 3 2812
6212c6212
< github.com/docker/distribution/registry/storage/filereader.go:70.2,70.15 1 3218
---
> github.com/docker/distribution/registry/storage/filereader.go:70.2,70.15 1 2812
6222c6222
< github.com/docker/distribution/registry/storage/filereader.go:93.29,95.4 1 1026
---
> github.com/docker/distribution/registry/storage/filereader.go:93.29,95.4 1 1027

Check this URL for a simpler diff.

There are only 30 diffs, out of 7011 lines. Checking them in a bit more detail, it seems like a small difference in the way the coverage tool generates some of the numbers for comparison. Based on this, I think the diff is quite insignificant so we can opt for the faster, shorter way of running the test coverage.