Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide the range of commits where a disabled test is effectively disabled #4980

Open
huydhn opened this issue Mar 2, 2024 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@huydhn
Copy link
Contributor

huydhn commented Mar 2, 2024

In the current implementation, disabling a test or enabling it (via a GitHub issues) take effect globally across all trunk and PR jobs. The good thing about this approach is that disabling a test is trivial. However, enabling them is still a tricky business. A common scenario is that a forward fix will address the issue and close it, but it will cause the test to fail on PRs everywhere unless people do a rebase to pull in the fix. We see this happening many times like the recent pytorch/pytorch#114831, which is directly responsible for a large spike of force merges.

After chatting with @clee2000 on the topic, there are several potential ideas for this:

  • We can provide the range of commits where a disabled test is effectively disabled. If the base commit of a PR is within the range, the test will still be disabled even if the issue has been closed. This seems like the best option.
  • At a coarse grain, we might be able to version the entire disabled tests JSON file. For example, a PR that has an older base commit will use an older version of the JSON file with the test still disabled

The same solution could also be applied to slow tests.

cc @clee2000

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: No status
Development

No branches or pull requests

1 participant