Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve how Test failures are summarized #9729

Closed
radeusgd opened this issue Apr 17, 2024 · 2 comments · Fixed by #9986
Closed

Improve how Test failures are summarized #9729

radeusgd opened this issue Apr 17, 2024 · 2 comments · Fixed by #9986
Assignees
@AdRiley
Labels
-libs Libraries: New libraries to be implemented x-chore Type: chore

Comments

@radeusgd
Copy link
Member

As suggested in a comment by @JaroslavTulach:

Printing up to ten of failed test cases at the end of test report would be useful!

We have a lot of tests and when I get the final summary (n failed tests) I still need to find out which of them failed. Often searching terminal history for [FAILED] doesn't work as the history isn't long enough. Then I need to re-run with ... | tee failed.txt and that slows everything down.

This indeed reminds us of a problem that I imagine every developer trying to figure out failed library tests has encountered - especially on the CI the log of tests that are run is HUGE and it is hard to browse it.

Locally we have the REPORT_ONLY_FAILED env var that allows to only print failures. I don't think it's viable to enable that flag on the CI by default because we do want to also see which tests are run.

As part of this task, one should design what is the best way to summarize the failed tests and implement it.

It may be the 'display up to 10 failed test cases at the end' as suggested, but it may be something else - the goal is to make it easy to find test failures when running a huge test suite.
@radeusgd radeusgd added x-chore Type: chore -libs Libraries: New libraries to be implemented labels Apr 17, 2024
@radeusgd
Copy link
Member Author

Possible solutions:

  1. reorder test summary so that failed groups are at the end
    i. cons:
    • with such reordering, one cannot display progress of tests being run but instead has to wait until all test suites finish which may take a long time; usually it is nice to see the progress of the test run
      • although that can be alleviated by adding a kind of progress bar, at least when running in terminal
    • big groups with just 1 test failure that appear at the end may still not be ideal
  2. display a short summary of failed tests, skipping the details (stack traces) and just listing the failures within each failed group (e.g. essentially repeat the output as would appear under REPORT_ONLY_FAILED, and possibly truncated to at most 10 cases as suggested.

@radeusgd radeusgd mentioned this issue Apr 17, 2024
Merged
5 tasks
@AdRiley AdRiley self-assigned this May 7, 2024
@AdRiley
Copy link
Member

AdRiley commented May 7, 2024

Make REPORT_ONLY_FAILED default to true -> REPORT_ALL_TESTS
Make sure xml report always contains all tests

@AdRiley AdRiley mentioned this issue May 17, 2024
Merged
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@AdRiley AdRiley

Labels
-libs Libraries: New libraries to be implemented x-chore Type: chore
Projects
Issues Board
Status: 🟢 Accepted
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Update test framework to default to only show failures enso-org/enso
2 participants
@radeusgd @AdRiley