Skip to content

Guard server telemetry against duplicate result updates per test UID#8192

Merged
Evangelink merged 4 commits into
mainfrom
copilot/fix-server-telemetry-multi-count
May 14, 2026
Merged

Guard server telemetry against duplicate result updates per test UID#8192
Evangelink merged 4 commits into
mainfrom
copilot/fix-server-telemetry-multi-count

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented May 13, 2026

Bug Fix

What was the bug?

Server-mode telemetry assumed result updates were effectively unique per test UID. If the same final outcome notification was emitted more than once, retries could be overcounted and request telemetry would report inflated result totals.

How did you fix it?

  • Result deduplication

    • Ignore repeated result updates when a test UID reports the same outcome it already has.
    • Continue counting only real outcome transitions as retries (for example, Failed -> Passed).
  • Telemetry semantics preserved

    • Passed / Failed totals still reflect the latest outcome per UID.
    • Retry counters now represent outcome changes rather than duplicate notifications.
  • Focused test coverage

    • Updated duplicate-passed coverage to assert that identical repeated outcomes do not increment retry counts.
    • Added duplicate-failed coverage.
    • Added coverage for an actual outcome change followed by a duplicate update to ensure only the first transition counts.

Example of the core change:

if (existingStatistics.HasPassed == hasPassed)
{
    return;
}

This makes duplicate Passed -> Passed or Failed -> Failed updates no-ops while preserving retry counting for genuine state changes.

Testing

  • Added/updated targeted unit tests for:
    • duplicate passed events
    • duplicate failed events
    • outcome change followed by a duplicate event

Copilot AI requested review from Copilot and removed request for Copilot May 13, 2026 17:03
Copilot AI linked an issue May 13, 2026 that may be closed by this pull request
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copilot AI requested review from Copilot and removed request for Copilot May 13, 2026 17:09
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copilot AI requested review from Copilot and removed request for Copilot May 13, 2026 17:10
Co-authored-by: Evangelink <11340282+Evangelink@users.noreply.github.com>
Copilot AI requested review from Copilot and removed request for Copilot May 13, 2026 17:11
Copilot AI changed the title [WIP] Fix server telemetry to guard against double counting results Guard server telemetry against duplicate result updates per test UID May 13, 2026
Copilot AI requested a review from Evangelink May 13, 2026 17:11
@Evangelink Evangelink marked this pull request as ready for review May 13, 2026 17:45
Copilot AI review requested due to automatic review settings May 13, 2026 17:45
Copy link
Copy Markdown
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Server-mode telemetry treated every result update as a distinct event, so duplicate final-outcome notifications for the same test UID inflated retry counters. This PR adds a short-circuit in AddOrUpdateTestNodeStateStatistics so only genuine outcome transitions (e.g. Failed → Passed) are counted as retries, and updates/extends the unit tests to cover duplicate Passed, duplicate Failed, and "transition then duplicate" scenarios.

Changes:

  • Skip statistics update when the new outcome matches the existing HasPassed value.
  • Adjust existing duplicate-passed test expectation (TotalPassedRetries 1 → 0) and rename the same-UID test to clarify it now exercises an outcome change.
  • Add new tests for duplicate failed events and for a duplicate event following an outcome change.
Show a summary per file
File Description
src/Platform/Microsoft.Testing.Platform/ServerMode/JsonRpc/PerRequestServerDataConsumerService.cs Early-return when the incoming outcome equals the stored one to avoid double-counting retries.
test/UnitTests/Microsoft.Testing.Platform.UnitTests/ServerMode/ServerDataConsumerServiceTests.cs Updates expectations for duplicate-passed test, renames the outcome-change test, and adds duplicate-failed and transition-then-duplicate tests.

Copilot's findings

  • Files reviewed: 2/2 changed files
  • Comments generated: 0

@Evangelink Evangelink enabled auto-merge (squash) May 14, 2026 07:38
@Evangelink Evangelink disabled auto-merge May 14, 2026 07:42
@Evangelink Evangelink merged commit 319bcc3 into main May 14, 2026
22 checks passed
@Evangelink Evangelink deleted the copilot/fix-server-telemetry-multi-count branch May 14, 2026 07:42
return;
}

if (existingStatistics.HasPassed == hasPassed)
Copy link
Copy Markdown
Member

@Youssef1313 Youssef1313 May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see why this is correct. Old code makes more sense to me.

If we receive two results, even if it's for the same test node, we should count it as two results.

It also doesn't make sense to gate the double counting under HasPassed == hasPassed.

If I have a folded test with two cases, imagine these two scenarios:

  • First case passed, second case failed -> we count it twice.
  • The two cases passed -> we count it once.

This doesn't make sense to me. I feel like the original issue is simply not correct and should just have been closed without making a change.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Server telemetry can multicount results

4 participants