feat: add telemetry scheduler #1107
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## feat/transport-envelope #1107 +/- ##
===========================================================
- Coverage 86.64% 86.03% -0.61%
===========================================================
Files 60 58 -2
Lines 6894 7080 +186
===========================================================
+ Hits 5973 6091 +118
- Misses 739 795 +56
- Partials 182 194 +12 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
giortzisg
force-pushed
the
feat/telemetry-scheduler
branch
4 times, most recently
from
1ecf9c6 to
d998585
Compare
giortzisg
force-pushed
the
feat/telemetry-scheduler
branch
from
6557a90 to
2b718b0
Compare
| firstItem := items[0] | ||
|
|
||
| header := &EnvelopeHeader{ | ||
| EventID: firstItem.GetEventID(), | ||
| SentAt: time.Now(), | ||
| Trace: firstItem.GetDynamicSamplingContext(), | ||
| } |
There was a problem hiding this comment.
Not sure if this works would work in the future for batching spans. For logs it's ok, since we get the trace id from each log item.
There was a problem hiding this comment.
If we would send the DSC as span attributes then you would be able to batch spans in any way you want, not necessarily group them by trace id always.
@cleptric is this planned?
There was a problem hiding this comment.
No plans at the moment. The main reason we batch spans by trace id is to enforce a certain traffic pattern in the ingestion segment buffer when it comes to insertion performance.
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
![seer-by-sentry[bot]](https://avatars.githubusercontent.com/in/801464?s=60&v=4){kind=small}
| func (e *Event) GetDynamicSamplingContext() map[string]string { | ||
| trace := make(map[string]string) | ||
| if dsc := e.sdkMetaData.dsc; dsc.HasEntries() { | ||
| for k, v := range dsc.Entries { | ||
| trace[k] = v | ||
| } | ||
| } | ||
| return trace | ||
| } | ||
|
|
||
| // TODO: Event.Contexts map[string]interface{} => map[string]EventContext, |
There was a problem hiding this comment.
Potential bug: When the telemetry buffer is enabled, the new Event.ToEnvelopeItem() method fails to include attachments from event.Attachments, causing them to be dropped from the final envelope.
-
Description: When the telemetry buffer feature is enabled, event attachments are silently lost. The new
Event.ToEnvelopeItem()method, which is called when processing buffered items, only converts the event itself into an envelope item and neglects to process theevent.Attachmentsfield. This is a regression from the previousEvent.ToEnvelope()implementation, which explicitly iterated over attachments and added them to the envelope. This results in silent data loss for any events that have attachments, both in the main scheduler path and the fallback path. -
Suggested fix: Modify
Event.ToEnvelopeItem()to handle attachments. It should create and returnprotocol.NewAttachmentItemfor each attachment found in theevent.Attachmentsslice, adding them to the envelope alongside the event item. This restores the functionality of the removedEvent.ToEnvelope()method.
severity: 0.85, confidence: 0.98
Did we get this right? 👍 / 👎 to inform future reviews.
internal/protocol/envelope.go
Outdated
| firstItem := items[0] | ||
|
|
||
| header := &EnvelopeHeader{ | ||
| EventID: firstItem.GetEventID(), |
There was a problem hiding this comment.
I'm not sure how it happens but when we send logs this is empty string so we get 400 from Relay, please fix it.
There was a problem hiding this comment.
Will continue the review on Monday.
Please add more debug logs to the transport as well (other PR I guess but here is also fine).
When testing manually I expect to see logs like
[Sentry] 2025/10/10 16:42:10 Sending info event [599f87ae22b848d7997cad821eb0fa8c] to o<x>.ingest.us.sentry.io project: <project>
[Sentry] 2025/10/10 16:42:10 Buffer flushed successfully.
and with the new transport + buffer I don't see anything and think stuff is not working.
Now it seems useless but when you get bug reports this is very useful info to have.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
This comment was marked as outdated.
This comment was marked as outdated.
Sorry, something went wrong.
| event.Logs = logs | ||
| l.client.CaptureEvent(event, nil, nil) | ||
| l.client.Transport.SendEvent(event) | ||
| //l.client.CaptureEvent(event, nil, nil) |
There was a problem hiding this comment.
Bug: Event Processing Bypassed in Debug Code
The processEvent function now calls l.client.Transport.SendEvent(event), bypassing the full event processing pipeline (event processors, sampling, BeforeSend callbacks, and scope application). The commented CaptureEvent line suggests this was debug code accidentally committed.
3 participants
Description
Add telemetry scheduler implementation and integrate with the client. The current PR also includes changes on the transport and envelope layer.
The proper solution for batching under the scheduler is to abstract each event type, so that it knows how to convert itself to an envelope item (noted as EnvelopeItemConvertible in the PR), and then the scheduler would just create envelopes using
[]EnvelopeItemConvertible. This partially fixes the issue, since currently everything is anchored around theEventtype. We should also abstract all event types to implementEnvelopeItemConvertiblein the future.Notes
EnvelopeConvertibletoEnvelopeItemConvertibleintroduced in feat: add new envelope transport #1094. For the sake of clarity I added the changes here and not on the other PR. Same applies withToEnvelopeEnvelopeHeader. Left some comments on the implementation.Issues