Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] TimeSeriesSortedSourceOperatorTests testRandom failing #108727

Closed
mosche opened this issue May 16, 2024 · 2 comments · Fixed by #108751
Closed

[CI] TimeSeriesSortedSourceOperatorTests testRandom failing #108727

mosche opened this issue May 16, 2024 · 2 comments · Fixed by #108751
Assignees
@dnhatn
Labels
low-risk An open issue or test failure that is a low risk to future releases :StorageEngine/Metrics You know, for Metrics Team:StorageEngine >test-failure Triaged test failures from CI

Comments

@mosche
Copy link
Contributor

mosche commented May 16, 2024

This was 8.14, but I can reproduce on main as well.

Build scan:
https://gradle-enterprise.elastic.co/s/tzsokr3ev4ibg/tests/:x-pack:plugin:esql:compute:test/org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests/testRandom

Reproduction line:

./gradlew ':x-pack:plugin:esql:compute:test' --tests "org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests.testRandom" -Dtests.seed=CDE0189DA184AA90 -Dtests.locale=de-AT -Dtests.timezone=Asia/Kabul -Druntime.java=21

Applicable branches:
8.14, main

Reproduces locally?:
Yes

Failure history:
Failure dashboard for org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests#testRandom

Failure excerpt:

org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.

  at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:67)
  at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:166)
  at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:160)
  at org.elasticsearch.compute.operator.OperatorTestCase.runDriver(OperatorTestCase.java:318)
  at org.elasticsearch.compute.operator.OperatorTestCase.runDriver(OperatorTestCase.java:277)
  at org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests.testRandom(TimeSeriesSortedSourceOperatorTests.java:172)
  at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
  at java.lang.reflect.Method.invoke(Method.java:580)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: java.util.concurrent.TimeoutException: Timeout waiting for task.

    at org.elasticsearch.action.support.PlainActionFuture$Sync.get(PlainActionFuture.java:258)
    at org.elasticsearch.action.support.PlainActionFuture.get(PlainActionFuture.java:71)
    at org.elasticsearch.common.util.concurrent.FutureUtils.get(FutureUtils.java:65)
    at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:166)
    at org.elasticsearch.action.support.PlainActionFuture.actionGet(PlainActionFuture.java:160)
    at org.elasticsearch.compute.operator.OperatorTestCase.runDriver(OperatorTestCase.java:318)
    at org.elasticsearch.compute.operator.OperatorTestCase.runDriver(OperatorTestCase.java:277)
    at org.elasticsearch.compute.lucene.TimeSeriesSortedSourceOperatorTests.testRandom(TimeSeriesSortedSourceOperatorTests.java:172)
    at jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
    at java.lang.reflect.Method.invoke(Method.java:580)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
    at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
    at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
    at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
    at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
    at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
    at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
    at org.junit.rules.RunRules.evaluate(RunRules.java:20)
    at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
    at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
    at java.lang.Thread.run(Thread.java:1583)
@mosche mosche added the >test-failure Triaged test failures from CI label May 16, 2024
@elasticsearchmachine elasticsearchmachine added needs:triage Requires assignment of a team area label needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 16, 2024
@mosche mosche added the :Analytics/Compute Engine Analytics in ES|QL label May 16, 2024
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label May 16, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytical-engine (Team:Analytics)

@elasticsearchmachine elasticsearchmachine removed the needs:triage Requires assignment of a team area label label May 16, 2024
@dnhatn dnhatn self-assigned this May 16, 2024
@dnhatn dnhatn added :StorageEngine/Metrics You know, for Metrics low-risk An open issue or test failure that is a low risk to future releases and removed :Analytics/Compute Engine Analytics in ES|QL needs:risk Requires assignment of a risk label (low, medium, blocker) labels May 16, 2024
@elasticsearchmachine elasticsearchmachine added Team:StorageEngine and removed Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) labels May 16, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-storage-engine (Team:StorageEngine)

@dnhatn dnhatn mentioned this issue May 17, 2024
Merged
dnhatn added a commit that referenced this issue May 17, 2024
@dnhatn
Fix thread reinitialize in time-series source operator (#108751)
a2ab2fe 
Currently, we only reinitialize the Lucene internal of the new top if 
the tsid changes. This isn't enough. We should always ensure the new top
is reinitialized if necessary, regardless of tsid.

Closes #108727
dnhatn added a commit to dnhatn/elasticsearch that referenced this issue May 19, 2024
@dnhatn
Fix thread reinitialize in time-series source operator (elastic#108751)
0ac68c1 
Currently, we only reinitialize the Lucene internal of the new top if 
the tsid changes. This isn't enough. We should always ensure the new top
is reinitialized if necessary, regardless of tsid.

Closes elastic#108727
@dnhatn dnhatn mentioned this issue May 19, 2024
Merged
elasticsearchmachine pushed a commit that referenced this issue May 19, 2024
@dnhatn
Fix thread reinitialize in time-series source operator (#108751) (#10…
93f6b31 
…8813)

Currently, we only reinitialize the Lucene internal of the new top if 
the tsid changes. This isn't enough. We should always ensure the new top
is reinitialized if necessary, regardless of tsid.

Closes #108727
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dnhatn dnhatn

Labels
low-risk An open issue or test failure that is a low risk to future releases :StorageEngine/Metrics You know, for Metrics Team:StorageEngine >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix thread reinitialize in time-series source operator dnhatn/elasticsearch
3 participants
@mosche @dnhatn @elasticsearchmachine