You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the current implementation, disabling a test or enabling it (via a GitHub issues) take effect globally across all trunk and PR jobs. The good thing about this approach is that disabling a test is trivial. However, enabling them is still a tricky business. A common scenario is that a forward fix will address the issue and close it, but it will cause the test to fail on PRs everywhere unless people do a rebase to pull in the fix. We see this happening many times like the recent pytorch/pytorch#114831, which is directly responsible for a large spike of force merges.
After chatting with @clee2000 on the topic, there are several potential ideas for this:
We can provide the range of commits where a disabled test is effectively disabled. If the base commit of a PR is within the range, the test will still be disabled even if the issue has been closed. This seems like the best option.
At a coarse grain, we might be able to version the entire disabled tests JSON file. For example, a PR that has an older base commit will use an older version of the JSON file with the test still disabled
The same solution could also be applied to slow tests.
In the current implementation, disabling a test or enabling it (via a GitHub issues) take effect globally across all trunk and PR jobs. The good thing about this approach is that disabling a test is trivial. However, enabling them is still a tricky business. A common scenario is that a forward fix will address the issue and close it, but it will cause the test to fail on PRs everywhere unless people do a rebase to pull in the fix. We see this happening many times like the recent pytorch/pytorch#114831, which is directly responsible for a large spike of force merges.
After chatting with @clee2000 on the topic, there are several potential ideas for this:
The same solution could also be applied to slow tests.
cc @clee2000
The text was updated successfully, but these errors were encountered: