Repository navigation
OneDS C++ SDK retries already-ingested iOS events, causing duplicate telemetry records #1542
Description
Describe your environment
- Client: Microsoft Teams for iOS
- Platform: iOS and iPadOS
- Device scope: Not isolated to a specific device model
- OS scope: Not isolated to a specific iOS or iPadOS version
- App scope: Multiple Microsoft Teams iOS versions
- OneDS SDK scope: Multiple versions from
3.4.295.1through3.10.173.1 - Current OneDS SDK:
3.10.173.1 - Current telemetry SDK field:
EVT-iOS-C++-No-3.10.173.1 - Event source:
OneCollector - Offline storage: OneDS persistent SQLite cache
- Transmission profile: Real-time
The issue is broadly distributed across the Teams iOS population. It is not isolated to one device model, OS version, app version, telemetry event, or OneDS SDK version.
One concrete affected environment is:
Device: iPhone18,2
OS: iOS 27.0.1
Teams version: 1417/8.17.77.2026172901
OneDS SDK: 3.10.173.1
Telemetry SDK field: EVT-iOS-C++-No-3.10.173.1
Event source: OneCollector
The same duplicate-delivery pattern was observed with multiple OneDS SDK versions:
3.10.173.1
3.10.40.1
3.9.324.1
3.9.309.1
3.9.267.1
3.9.212.1
3.8.249.1
3.6.187.1
3.5.200.1
3.5.189.1
3.5.148.1
3.5.127.1
3.5.25.1
3.4.295.1
We checked the current main branch of microsoft/cpp_client_telemetry. The relevant retry behavior is still present:
- Confirmed accepted requests delete records from offline storage.
- Network failures release records back to offline storage.
- Aborted requests release records back to offline storage.
- We could not find an idempotency mechanism that prevents an event already accepted by Collector from being ingested again if the client does not receive or process the acknowledgement.
Relevant current-main files:
- https://github.com/microsoft/cpp_client_telemetry/blob/main/lib/system/TelemetrySystem.cpp
- https://github.com/microsoft/cpp_client_telemetry/blob/main/lib/http/HttpResponseDecoder.cpp
- https://github.com/microsoft/cpp_client_telemetry/blob/main/lib/offline/StorageObserver.cpp
Steps to reproduce
The issue appears when an upload has an ambiguous result—for example, Collector accepts the request, but the client does not receive or process the HTTP success response.
A likely reproduction flow is:
- Initialize the OneDS C++ SDK with persistent SQLite offline storage.
- Log a telemetry event.
- Allow OneDS to begin uploading the event to Collector.
- After the request has reached Collector but before the HTTP success callback is processed, interrupt the response path by doing one of the following:
- terminate the application;
- call
pauseTransmission()or otherwise cancel active requests; - interrupt the network connection after the request is sent.
- Relaunch the application or restore network connectivity.
- Allow OneDS to process its persistent offline cache.
- Query the destination telemetry table using the event timestamp, sequence number, session ID, and pipeline record ID.
- Observe that the same serialized event was ingested multiple times with different pipeline ingestion timestamps.
The following production query demonstrates one occurrence:
scenario
| where Scenario_Step == "STOP"
| where Scenario_ID == "11FC326D-9FC3-47E0-B242-C7E2D592EBE4"
| project
Scenario_ID,
Scenario_Step,
Scenario_Status,
Scenario_ScenarioTimeTaken,
EventInfo_Time,
EventInfo_Sequence,
Session_Id,
PipelineInfo__550_55Id,
EventInfo_SdkVersion,
PipelineInfo_IngestionTime
| order by PipelineInfo_IngestionTime ascThe affected event has the following identity:
Scenario_ID: 11FC326D-9FC3-47E0-B242-C7E2D592EBE4
Scenario_Step: STOP
Scenario_Status: OK
Scenario_ScenarioTimeTaken: 6527.508974
EventInfo_Time: 2026-09-29 15:12:43.272 UTC
EventInfo_Sequence: 632
OneDS SDK: 3.10.173.1
Event source: OneCollector
The same event was ingested at the following times:
2026-09-29 16:10:29.882 UTC
2026-09-29 18:47:54.833 UTC
2026-09-29 20:02:04.765 UTC
2026-09-29 20:25:11.776 UTC
2026-09-29 21:08:53.200 UTC
All five rows have the same:
Scenario_IDEventInfo_TimeEventInfo_SequenceSession_IdPipelineInfo__550_55IdEventInfo_SdkVersion- status
- duration
- event payload
Only PipelineInfo_IngestionTime differs.
The identical sequence number, session ID, and pipeline record ID indicate that the application created the event once and that the same serialized OneDS record was delivered repeatedly.
What is the expected behavior?
One logical client event should produce one logical record in the destination telemetry table.
If OneDS retries an upload because the upload outcome is ambiguous, one of the following should prevent duplicate ingestion:
- Collector should deduplicate retries using a stable event or record identifier.
- The upload protocol should provide an idempotency key that remains stable across retries.
- The SDK and Collector should use another mechanism that prevents an already-accepted event from creating a second logical record.
Retries are expected and necessary for telemetry reliability. However, an event already accepted by Collector should not be counted multiple times in downstream telemetry.
What is the actual behavior?
The same serialized OneDS event is ingested multiple times.
Each copy retains the same:
- event timestamp;
- sequence number;
- session ID;
- client event ID;
- pipeline record ID;
- SDK version;
- event source;
- payload.
Each copy receives a different PipelineInfo_IngestionTime.
In the concrete example, the same event was ingested five times over approximately 298 minutes.
This inflates:
- telemetry event volume;
- success and failure counts;
- reliability metrics;
- latency metrics;
- usage metrics;
- telemetry ingestion cost.
Additional context
Production impact
A five-minute production sample from September 29, 2026 contained:
| Metric | Result |
|---|---|
| Logical telemetry events evaluated | 34,725,838 |
| Logical events affected by duplicate delivery | 346,712 |
| Extra ingested rows | 366,734 |
| Approximate affected logical-event rate | 1.0% |
| Maximum observed copies of one event | 7 |
This indicates a systemic issue rather than an isolated event-instrumentation problem.
Impact by OneDS SDK version
| OneDS SDK version | Affected logical events | Approximate affected rate |
|---|---|---|
3.10.40.1 |
208,469 | 0.96% |
3.10.173.1 |
129,372 | 1.07% |
3.8.249.1 |
2,193 | 0.91% |
3.9.324.1 |
1,965 | 0.72% |
3.6.187.1 |
1,251 | 1.57% |
3.9.309.1 |
1,223 | 0.83% |
3.9.267.1 |
988 | 0.84% |
3.9.212.1 |
825 | 0.73% |
The presence of duplicates across multiple SDK generations indicates that this is not a regression isolated to SDK 3.10.173.1.
Suspected failure mode
The observed behavior is consistent with at-least-once delivery:
- OneDS reserves a record from offline storage.
- OneDS sends the record to Collector.
- Collector accepts and persists the event.
- The response is lost, interrupted, or canceled before the SDK processes the success acknowledgement.
- The SDK classifies the request as a network failure or aborted request.
- The SDK releases the record back to offline storage.
- OneDS retries the same serialized record later.
- Collector ingests the record again without deduplicating it.
The relevant current-main routing is:
httpDecoder.eventsAccepted
>> storage.deleteRecords
>> stats.onUploadSuccessful
>> tpm.eventsUploadSuccessful;
httpDecoder.temporaryNetworkFailure
>> storage.releaseRecords
>> stats.onUploadFailed
>> tpm.eventsUploadFailed;
httpDecoder.requestAborted
>> storage.releaseRecords
>> stats.onUploadFailed
>> tpm.eventsUploadAborted;This behavior avoids losing telemetry when the client cannot confirm success. However, without end-to-end idempotency, it produces duplicate ingestion when Collector accepted the original request but the acknowledgement was not received.
Activity
bmehta001 commentedon Oct 1, 2026
Hi @nischith-cp , Can you please try the latest 3.10.267.1 and confirm if you still observe this error? I had included a lot of fixes on the Apple side and SQLite side to prevent race conditions and duplicate sends; you can also point CoPilot CLI at this codebase and discuss with it to confirm if it fixes your errors before you test
Priyanka-Dobhal commentedon Oct 8, 2026
Hi bmehta001, checked this via CoPilot CLI and looks like the issue will not be resolved completely even after updating to 3.10.267.1
bmehta001 commentedon Oct 12, 2026
Hi Priyanka-Dobhal,
Thanks for taking a look. 3.10.267.1 and later versions address some of the runtime and storage issues that could lead to duplicate sends in practice (e.g., crashes, hangs, and retrying failed storage flushes), but that would require deployment to measure the difference, and you are right that the latest SDK does not fix all of the underlying issues. When Collector accepts an upload but the client never receives or processes the acknowledgement, the client cannot reliably distinguish that from an upload that was never accepted. Retrying preserves the chance of delivery but can result in duplicates unless the receiving pipeline deduplicates them.
You can work with CoPilot to try changing existing settings/config within the SDK like max teardown upload time to potentially decrease duplicates during graceful teardown, assuming the tradeoffs are acceptable to you. The amount of improvement would need to be measured. For example, setting this to 2 seconds allows the SDK to spend approximately that long attempting uploads before proceeding with cancellation and the rest of shutdown; it does not cap the total teardown time at 2 seconds. It also does not change the cancellation behavior of pauseTransmission(), does not apply when transmission is already paused, and cannot help if the app is abruptly terminated.
Some of the design decisions in the 1DS SDK have a tradeoff between avoiding duplicate sends and risking dropping events prematurely. Presented with this tradeoff, some 1DS consumers would prefer to deal with the issue of deduplicating events server-side rather than risk missing vital events altogether. So, we intentionally lean towards being conservative with assuming events have been sent successfully, which can potentially result in duplicate events.
I would distinguish 1 from 3 here. For 1, the lost-acknowledgement case requires cooperation from Collector or the downstream pipeline to reliably prevent duplicate ingestion without risking dropping events. For 3, you are right that the Apple transport in 3.10.267.1 gives a pending cancellation precedence even over an otherwise error-free HTTP 200 completion, and that behavior is still present on main. Preserving a fully completed, error-free, protocol-valid success in that race looks like something worth investigating separately, without treating an incomplete response as proof that the events were accepted. Just seeing status 200 is not sufficient if the transfer subsequently fails.
For 2, adding an idempotency key in the SDK alone would not prevent duplicates; Collector or the downstream pipeline would need to recognize and enforce it. The SDK already includes an initialization epoch and a sequence number in serialized events, so it would be worth checking whether those fields reach your destination and can be used as a suitably scoped deduplication key before adding another identifier. We would also need to confirm the receiving pipeline's actual deduplication behavior with the Collector team rather than infer it solely from this SDK.
If you or Nishchith C P (@nishchith-cp) are interested in a more robust fix, can you please work with CoPilot on this issue, tell CoPilot that the fix be fully backwards-compatible and tested, and point it to cpp_client_telemetry/docs/maintainer-onboarding.md at main · microsoft/cpp_client_telemetry and ask it to follow the relevant testing instructions. Shared storage changes should be tested on Windows and WSL/Linux, and Apple networking/lifecycle changes should also be tested on macOS/iOS Simulator, with physical-device testing for the backgrounding and network-interruption behavior where needed. CI can supplement local testing, but please check what actually ran rather than assume it covers every gap. You can try integrating the SDK into your app to test locally and ensure events are received as expected on the backend your team uses, as well as manually review and have CoPilot CLI and the built-in GitHub CoPilot review the PR in a loop until you find no further issues. Then, ping me or message General once you have a PR, and I will also do further testing on each necessary platform.
This way, if you are the one creating the PR rather than me, I can approve it rather than having to find someone else with approval permissions to then approve my PR.
You can try addressing one issue at a time in each PR and comment here to let everyone know about the PRs you have created, or fixes/approaches you have ruled out. Reducing avoidable duplicate sends in the SDK and working with the receiving pipeline on deduplication can be pursued separately; we should not assume an SDK-only change will eliminate all duplicates.
bmehta001 commentedon Nov 5, 2026
#1556 should help skip some duplicate events