Skip to content

end of output from runner getting lost (regression on gitlab.com around 20th April 2020)

Summary

The end of stdout from a job running on a runner is lost.

Steps to reproduce

Running the CI on https://gitlab.com/openid/conformance-suite

View the output log for deployment_test_staging

Scroll to end and verify if "All tests ran to completion." line is present as the last line of output.

Example Project

https://gitlab.com/openid/conformance-suite

What is the current bug behavior?

Significant amounts of output are missing, possibly more than 100 lines.

This particular script produces approx 2000 lines of output in it's final second or two, which may be a factor.

The text is missing in the raw log as well, so this isn't a web frontend issue.

What is the expected correct behavior?

All output is there, as it was prior to 20th April 2020.

Relevant logs and/or screenshots

Good run on 19th April ~6AM UTC: https://gitlab.com/openid/conformance-suite/-/jobs/517346468

Good run on 20th April ~6AM UTC: https://gitlab.com/openid/conformance-suite/-/jobs/517868088

Bad run on 21st April ~6AM UTC: https://gitlab.com/openid/conformance-suite/-/jobs/519538349

Bad run on 22nd April ~6AM UTC: https://gitlab.com/openid/conformance-suite/-/jobs/521272810

Bad run today (9th May ~6AM UTC): https://gitlab.com/openid/conformance-suite/-/jobs/545117171

Output of checks

This bug happens on GitLab.com

Workaround

A possible workaround for this issue is to add a time.sleep(20) at the end of the job (notice that in the linked file the sleep time was set to 10, but it looks like it's working more consistently with 20.

Current status

  • For Docker executor:
  • For issues on Windows:
    • Still trying to get more feedback from users regarding the exact issue, most of the reports so far are lacking details that allow for more detailed investigation.
Edited by Pedro Pombeiro