Code coverage for integration tests
Problem to solve
Currently, we are not gathering code coverage for our automated test (#199 (closed)). We don't know how much of our code is untested.
Proposal
Use nyc
in our test suite to gather coverage. This might require us to redesign the activation of our extension or discuss some possible fixes with the vscode team. The redesign might bring a benefit of VS Code starting faster for our users.
Further details
This is a complex task.
There is a rough guideline for adding nyc
to an extension project - GitHub issue in vscode
And there is a good example in the microsoft/vscode-js-debug: The VS Code JavaScript debugger project.
src/test/testRunner.ts
function setupCoverage() {
const NYC = require('nyc');
const nyc = new NYC({
cwd: join(__dirname, '..', '..', '..'),
exclude: ['**/test/**', '.vscode-test/**'],
reporter: ['text', 'html'],
all: true,
instrument: true,
hookRequire: true,
hookRunInContext: true,
hookRunInThisContext: true,
});
nyc.reset();
nyc.wrap();
return nyc;
}
// ...
finally {
if (nyc) {
nyc.writeCoverageFile();
nyc.report();
}
}
The issue with this solution is that it only gathers coverage for files that have been required AFTER this setupCoverage()
has been called.
This works for the vscode-js-debug
extension, because they don't immediately activate the extension. (Documentation on activation events).
- vscode-js-debug - package.json
"activationEvents": [ "onLanguage:json", "onDebugDynamicConfigurations", "onDebugInitialConfigurations", "onDebugResolve:node", "onDebugResolve:extensionHost", "onDebugResolve:chrome" ],
- gitlab-workflow - package.json
"activationEvents": [ "*" ],
By the time we initialize nyc
the whole extension with all dependencies has been already required.
I've spent 6 hours finding the problem and then investigating a solution. The most promising solution might be
Change the activation events
I've tried the following events:
"onCommand:gl.showIssuesAssignedToMe",
"onCommand:gl.showMergeRequestsAssignedToMe",
"onCommand:gl.setToken",
"onCommand:gl.removeToken",
"onCommand:gl.openActiveFile",
"onCommand:gl.copyLinkToActiveFile",
"onCommand:gl.openCurrentMergeRequest",
"onCommand:gl.openCreateNewIssue",
"onCommand:gl.openCreateNewMR",
"onCommand:gl.openProjectPage",
"onCommand:gl.openCurrentPipeline",
"onCommand:gl.pipelineActions",
"onCommand:gl.issueSearch",
"onCommand:gl.mergeRequestSearch",
"onCommand:gl.compareCurrentBranch",
"onCommand:gl.createSnippet",
"onCommand:gl.validateCIConfig",
"onCommand:gl.refreshSidebar",
"onCommand:gl.showRichContent",
"onView:issuesAndMrs",
"workspaceContains:.git"
Note: the .git
activation event is necessary because we want to show the status bar almost immediately
This solution produced results but with very fragile test infrastructure code.