feat(ACI): Send historical data to seer on update #102934
Conversation
github-actions
bot
added
the
Scope: Backend
| "id": self.data_condition_group.id, | ||
| "organizationId": self.organization.id, |
There was a problem hiding this comment.
These aren't sent by the front end so I wanted this to be the same. Especially for the ids, it doesn't make sense that we'd be sending these on creation.
| @@ -553,6 +551,379 @@ def test_transaction_dataset_deprecation_multiple_data_sources(self) -> None: | |||
| ): | |||
| validator.save() | |||
|
|
|||
|
|
|||
| class TestMetricAlertsUpdateDetectorValidator(TestMetricAlertsDetectorValidator): | |||
| def test_update_with_valid_data(self) -> None: | |||
There was a problem hiding this comment.
We didn't have a simple update test case so I added one
| raise DetectorException( | ||
| f"Could not create detector, data condition {dcg_id} not found or too many found." | ||
| ) | ||
| # use setattr to avoid saving the models until the Seer call has successfully finished, |
There was a problem hiding this comment.
This is the same as we do it for alert rules https://github.com/getsentry/sentry/blob/master/src/sentry/seer/anomaly_detection/store_data.py#L122
ceorourke
force-pushed
the
ceorourke/send-historical-data-to-seer-on-update
branch
from
c4fdd12 to
2adcc69
Compare
❌ 3 Tests Failed:
View the top 3 failed test(s) by shortest run time
To view more test analytics, go to the Test Analytics Dashboard |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
![sentry[bot]](https://avatars.githubusercontent.com/in/12637?s=60&v=4){kind=small}
| resolution=timedelta(seconds=data_source.get("resolution", snuba_query.resolution)), | ||
| environment=data_source.get("environment", snuba_query.environment), | ||
| event_types=data_source.get("event_types", [event_type for event_type in event_types]), | ||
| ) |
There was a problem hiding this comment.
Bug: update_detector_data receives a single data source dict instead of a {"data_sources": [...]} structure, preventing snuba_query updates.
Severity: CRITICAL | Confidence: 0.95
🔍 Detailed Analysis
When update_detector_data is invoked at src/sentry/incidents/metric_issue_detector.py:249, it receives validated_data_source, which is a single dictionary. However, the update_detector_data function expects a dictionary containing a "data_sources" key with a list of data sources. This mismatch causes the internal logic to skip updating the snuba_query object's fields. Consequently, when a dynamic detector's snuba query is updated, the old query, aggregate, and event types are sent to Seer instead of the new values, leading to anomaly detection operating on incorrect metrics.
💡 Suggested Fix
Modify the call to update_detector_data at src/sentry/incidents/metric_issue_detector.py:249 to pass {"data_sources": [validated_data_source]} instead of validated_data_source directly, aligning with the expected input structure.
🤖 Prompt for AI Agent
Review the code at the location below. A potential bug has been identified by an AI
agent.
Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not
valid.
Location: src/sentry/incidents/metric_issue_detector.py#L249
Potential issue: When `update_detector_data` is invoked at
`src/sentry/incidents/metric_issue_detector.py:249`, it receives
`validated_data_source`, which is a single dictionary. However, the
`update_detector_data` function expects a dictionary containing a `"data_sources"` key
with a list of data sources. This mismatch causes the internal logic to skip updating
the `snuba_query` object's fields. Consequently, when a dynamic detector's snuba query
is updated, the old query, aggregate, and event types are sent to Seer instead of the
new values, leading to anomaly detection operating on incorrect metrics.
Did we get this right? 👍 / 👎 to inform future reviews.
mifu67
left a comment
mifu67
left a comment
There was a problem hiding this comment.
The update logic itself looks good; just a question about when we should be updating.
src/sentry/seer/anomaly_detection/store_data_workflow_engine.py
Outdated
Show resolved
Hide resolved
|
|
||
| # Handle a dynamic detector's snuba query changing | ||
| if instance.config.get("detection_type") == AlertRuleDetectionType.DYNAMIC: | ||
| if snuba_query.query != data_source.get( |
There was a problem hiding this comment.
Do we need to check other snuba query fields as well (like timeWindow)? Should we just resend the data every time we update a dynamic detector?
There was a problem hiding this comment.
Good question, I will look into this for a follow up. I think every time we change it isn't necessary (e.g. it could just be changing the name of the detector) but we might be missing some cases we should be updating.
When a static or percent based detector is changed to become a dynamic detector we need to send Seer historical data for that detector so it can detect anomalies. Also when a dynamic detector's snuba query query or aggregate changes we need to update the data Seer has so it's detecting anomalies on the correct data. This PR also handles not updating the existing data if the call to Seer fails for any reason.