User acceptance tests are broken during gitlab CI
Recently (December 3th at noon) the CI pipeline started failing during the execution of the user acceptance tests.
As even old builds that previously succeed started failing, out conclusion is that this is an issue in the Gitlab infrastructure.
As a temporary workaround we set the user acceptance tests to be allowed to fail, in order to allow to pipeline to continue succeeding (MR !353 (merged)), but this is not a proper solution. It would consist on fixing once and for all the issue.
It's also possible that gitlab is behaving properly and such tests were never supposed to succed in the first place. In such scenario, we should try to fix the tests so that they run in the new gitlab configuration, whatever it is.
More investigation is needed.
Related to that, at the same time we noticed the issue, other users started reporting similar disrruptions in their pipelines. The link for it in the Gitlab support forum is: https://forum.gitlab.com/t/ci-cd-pipeline-is-pending/45958
This is how the complete job logs look like:
Running with gitlab-runner 13.6.0 (8fa89735)
on docker-auto-scale ed2dce3a
Resolving secrets
00:00
Preparing the "docker+machine" executor
00:26
Using Docker executor with image registry.gitlab.com/lightmeter/acceptance-tests-docker-image ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.gitlab.com/lightmeter/acceptance-tests-docker-image ...
Using docker image sha256:ea5b4750c5a2a45c6c29fe1daf1ac3703d00bc660c239b6974743214b03a092c for registry.gitlab.com/lightmeter/acceptance-tests-docker-image with digest registry.gitlab.com/lightmeter/acceptance-tests-docker-image@sha256:e901f951b9fc478eb60e96f7f0957c3828b5696098349ff1afd9edeeb4c9783b ...
Preparing environment
00:02
Running on runner-ed2dce3a-project-17017123-concurrent-0 via runner-ed2dce3a-srm-1607002136-581f1d8d...
Getting source from Git repository
00:04
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/lightmeter/controlcenter/.git/
Created fresh repository.
Checking out ac23abbf as bot/weblate...
Skipping Git submodules setup
Restoring cache
00:02
Checking cache for acceptance-test-2...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/17017123/acceptance-test-2
Successfully extracted cache
Downloading artifacts
00:01
Downloading artifacts for build (888517275)...
Downloading artifacts from coordinator... ok id=888517275 responseStatus=200 OK token=i9AjvKFk
Executing "step_script" stage of the job script
01:34
$ npm --prefix acceptance_tests link
npm WARN controlcenter-acceptance-tests@0.0.1 No repository field.
npm WARN controlcenter-acceptance-tests@0.0.1 No license field.
audited 128 packages in 1.146s
5 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
/builds/lightmeter/controlcenter/acceptance_tests/lib/node_modules/controlcenter-acceptance-tests -> /builds/lightmeter/controlcenter/acceptance_tests
$ npm --prefix acceptance_tests test
> controlcenter-acceptance-tests@0.0.1 test /builds/lightmeter/controlcenter/acceptance_tests
> node -c tests/*.js && gauge run specs/
Compatible version of plugin js not found. Installing plugin js...
.......................
audited 310 packages in 1.715s
2 packages are looking for funding
run `npm fund` for details
found 0 vulnerabilities
Successfully installed plugin 'js' version 2.3.14
Compatible version of plugin xml-report not found. Installing plugin xml-report...
........................
Successfully installed plugin 'xml-report' version 0.2.3
Installing required plugins.
........................
Successfully installed plugin 'screenshot' version 0.0.1
# Registration
## Registration fails ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔accepting popup with message: Please select an option for 'Most of my mail is' - see help for details
✔
## Registraton succeeds ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔ ✔
# logout
## logout success [DEPRECATED] gauge.screenshotFn will be removed soon, use gauge.customScreenshotWriter instead.
✘
Failed Step: Go to homepage
Specification: specs/2_logout.spec:6
Error Message: Error: Timed out
Stacktrace:
Error: Timed out
at Timeout.<anonymous> (/home/node/.gauge/plugins/js/2.3.14/src/test.js:44:23)
at listOnTimeout (internal/timers.js:551:17)
at processTimers (internal/timers.js:494:7)
# login
## login success ✔[DEPRECATED] gauge.screenshotFn will be removed soon, use gauge.customScreenshotWriter instead.
✘
Failed Step: Focus on field with placeholder "Email"
Specification: specs/3_login.spec:7
Error Message: Error: TextBox[placeholder="Email"] not found
Stacktrace:
Error: TextBox[placeholder="Email"] not found
at findElements (node_modules/taiko/lib/elementSearch.js:193:11)
at async findFirstElement (node_modules/taiko/lib/elementSearch.js:175:11)
at async node_modules/taiko/lib/taiko.js:1347:30
at async doActionAwaitingNavigation (node_modules/taiko/lib/doActionAwaitingNavigation.js:60:5)
at async module.exports.focus (node_modules/taiko/lib/taiko.js:1345:3)
at async module.exports.<computed> (node_modules/taiko/lib/taiko.js:3232:14)
at async Object.<anonymous> (tests/step_implementation.js:57:5)
# Datepicker
## Show insights ✔ ✔ ✔ ✔ ✔ ✔
Successfully generated html-report to => /builds/lightmeter/controlcenter/acceptance_tests/reports/html-report/index.html
Successfully generated xml-report to => /builds/lightmeter/controlcenter/acceptance_tests/reports/xml-report
Specifications: 4 executed 2 passed 2 failed 0 skipped
Scenarios: 5 executed 3 passed 2 failed 0 skipped
Total time taken: 1m25.978s
Updates are available. Run `gauge update -c` for more info.
npm ERR! Test failed. See above for more details.
Uploading artifacts for failed job
00:02
Uploading artifacts...
acceptance_tests/reports/xml-report/result.xml: found 1 matching files and directories
Uploading artifacts as "junit" to coordinator... ok id=888539069 responseStatus=201 Created token=zxSbZiEj
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1Edited by Leandro Santiago