Example test report (summary) for CIP releases

Issue Background

To make it easier for developers to judge if a release passes all tests it should pass, a report in executive summary style is required. The current reporting in SQUAD might do, but

• it's not clear which tests are allowed to fail (information, that the developer doesn't care about)
• hard to get an overview of tests that aren't allowed to fail (information, that the developer needs to known about)
• no report format available in SQUAD that says: "All ok, all tests passed", when a build passes all (required) tests

Issue description

This issues is about creating a mechanism, tool or script, that produces an overview/report (executive summary) for a defined build that was tagged as release (candidate), as announced with the [ANNOUNCED] tag on the mailing list. The report should show

• either that all required tests have passed
• or that some required tests have failed
• it would be nice if all test that have failed, but are allowed to fail are also shown (perhaps hidden behind expandable plus or something similar)

Acceptance criteria

• It is clear how to distinguish required tests from tests that are allowed to fail
• The format of the report has been agreed upon
• Functional Report/management summary creation for a defined build in SQUAD, containing
  • Overall Pass or fail statement
  • If fail statement: overview of required failed tests
  • Possibility to expand a list of failed tests, that are allowed to fail
• Report management summary has been tested on a previously announced CIP kernel release.
• Overall pass/fail message includes a disclaimer if there were incomplete test jobs
• Extend the cip-release-reports interface to be able to
  • Generate a single release report for a given group, project and build
  • Build release reports for all groups if the group options is omitted
  • Build release reports all project if the project options is omitted
  • Build release reports all builds if the build options is omitted