Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ResponseOps] rules runs recovered actions without ever running active actions #182888

Closed
pmuellr opened this issue May 7, 2024 · 1 comment · Fixed by #183646
Closed

[ResponseOps] rules runs recovered actions without ever running active actions #182888

pmuellr opened this issue May 7, 2024 · 1 comment · Fixed by #183646
Assignees
@doakalexi
Labels
bug Fixes for quality problems that affect the customer experience Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@pmuellr
Copy link
Member

pmuellr commented May 7, 2024

We've seen some cases where rules will run the recovery actions, but we're not seeing any of the active actions being run. Like they've been lost, or perhaps there weren't any active actions set up. But, that doesn't seem to be the case with some of the investigation we've done.

We believe this may be an issue with the "alert after x consecutive matches".

From Ying:

It is set to alert after 5 consecutive matches. If there were not 5 consecutive matches no alert would be created but we still track it as an active alert in the task manager state so if it recovers it would trigger a recovery notification ...

We probably need to update the logic to not consider it recovered if it hasn't met the threshold for an active alert

Ersin was able to repro this.
@pmuellr pmuellr added bug Fixes for quality problems that affect the customer experience Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) labels May 7, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@doakalexi doakalexi self-assigned this May 14, 2024
@doakalexi doakalexi mentioned this issue May 16, 2024
Merged
1 task
doakalexi added a commit that referenced this issue May 20, 2024
@doakalexi
[ResponseOps] rules runs recovered actions without ever running activ…
0f159b7 
…e actions (#183646)

Resolves #182888

## Summary

This PR resolves a bug where rules will run the recovery actions for a
delayed active alert.

### Checklist

- [ ] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios


### To verify

- Create a rule and set the `alert after x consecutive matches` to be
greater than 1. It may be helpful for testing to include recovered and
active actions.
- Make sure the delayed active alert recovers before hitting the
consecutive matches threshold.
- Verify that the rule does not send a recovery action and does not show
a recovered alert in the rule run table.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@doakalexi doakalexi

Labels
bug Fixes for quality problems that affect the customer experience Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
AppEx: ResponseOps - Execution & Connectors
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[ResponseOps] rules runs recovered actions without ever running active actions doakalexi/kibana
3 participants
@pmuellr @elasticmachine @doakalexi