Skip to content

Apply structured assertion messages (RFC 012) to AreEqual / AreNotEqual #8231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Evangelink wants to merge 19 commits into
base: main
Choose a base branch
from

Merge origin/main and resolve conflicts

1950cf2
Select commit
Loading
Failed to load commit list.
Open

Apply structured assertion messages (RFC 012) to AreEqual / AreNotEqual #8231

Merge origin/main and resolve conflicts
1950cf2
Select commit
Loading
Failed to load commit list.
Azure Pipelines / microsoft.testfx (Build Linux Debug) failed May 16, 2026 in 13m 49s

Build Linux Debug failed

3 errors / 0 warnings

Annotations

Check failure on line 175 in src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Linux Debug)

src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs#L175 

src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs(175,1): error : [ScopeWithSingleFailure_TestFails] [net11.0] Assertion failed. Expected condition to be true.
Expression 'AssertOutputMatchesRegex' failed for member 'ScopeWithSingleFailure_TestFails' at line 30 of file '/_/test/IntegrationTests/MSTest.Acceptance.IntegrationTests/SoftAssertionTests.cs'. Output of the test host is:
Command: /mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/WmODQ/SoftAssertionTests/bin/Release/net10.0/SoftAssertionTests --filter ScopeWithSingleFailure
====================
ExitCode: 2
====================
StandardOutput:
MSTest v4.3.0-ci (UTC 05/16/2026) [linux-x64 - .NET 10.0.8]
failed ScopeWithSingleFailure (92ms)
  Assertion failed. Expected values to be equal.
  expected: 1
  actual:   2
  Assert.AreEqual(1, 2)
    at Microsoft.VisualStudio.TestTools.UnitTesting.AssertScope.Dispose() in /_/src/TestFramework/TestFramework/Assertions/AssertScope.cs:72
    at UnitTest1.ScopeWithSingleFailure() in /mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/WmODQ/SoftAssertionTests/UnitTest1.cs:23
    at UnitTest1.ScopeWithSingleFailure() in /mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/WmODQ/SoftAssertionTests/UnitTest1.cs:20
    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Test run summary: Failed! - /mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/WmODQ/SoftAssertionTests/bin/Release/net10.0/SoftAssertionTests.dll (net10.0|x64)
  total: 1
  failed: 1
  succeeded: 0
  skipped: 0
  duration: 678ms
====================
StandardError:


actual: false

Assert.IsTrue(Regex.IsMatch(testHostResult.StandardOutput, pattern))

Check failure on line 454 in src/TestFramework/TestFramework/Assertions/Assert.Contains.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Linux Debug)

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs#L454 

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs(454,1): error : [TrxReport_WhenTestFails_ContainsExceptionInfoInOutput ("net10.0")] [net11.0] Assert.Contains failed. String '<?xml version="1.0" encoding="utf-8"?>
<TestRun id="fbcf16f0-b94c-456d-8c7a-784ad6a739ce" name="@b96b181dc000000 2026-05-16 14:51:28.3108826" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T14:51:27.9667132Z" queuing="2026-05-16T14:51:27.9667132Z" start="2026-05-16T14:51:27.9667132Z" finish="2026-05-16T14:51:28.3117671Z" />
  <TestSettings name="default" id="098aeb3d-18dd-4b7e-bebd-3d6d6cc019de">
    <Deployment runDeploymentRoot="_b96b181dc000000_2026-05-16_14_51_28.3108826" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="192c918b-a046-4527-a919-ecfb7a7b6d1b" testId="18f2dd82-fede-8c47-821a-9500badc6846" testName="FailingTest" computerName="b96b181dc000000" duration="00:00:00.0457987" startTime="2026-05-16T14:51:28.1256581+00:00" endTime="2026-05-16T14:51:28.2268957+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="192c918b-a046-4527-a919-ecfb7a7b6d1b">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 298
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 142
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 48
   at MSTestTrxReport.UnitTest1.FailingTest() in /mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/C9q2X/MSTestTrxReport/UnitTest1.cs:line 11
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="/mnt/vss/_work/1/s/artifacts/tmp/debug/testsuite/c9q2x/mstesttrxreport/bin/release/net10.0/mstesttrxreport.dll" id="18f2dd82-fede-8c47-821a-9500badc6846">
      <Execution id="192c918b-a046-4527-a919-ecfb7a7b6d1b" />
      <TestMethod codeBase="/mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/C9q2X/MSTestTrxReport/bin/Release/net10.0/MSTestTrxReport.dll" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="18f2dd82-fede-8c47-821a-9500badc6846" executionId="192c918b-a046-4527-a919-ecfb7a7b6d1b" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="b96b181dc000000" outcome="Error" timestamp="2026-05-16T14:51:28.3298436">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>' does not contain string 'Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;.'. 'substring' expression: '"Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;."', 'value' expression: 'trxContent'. <?xml version="1.0" encoding="utf-8"?>
<TestRun id="fbcf16f0-b94c-456d-8c7a-784ad6a739ce" name="@b96b181dc000000 2026-05-16 14:51:28.3108826" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T14:51:27.9667132Z" queuing="2026-05-16T14:51:27.9667132Z" start="2026-05-16T14:51:27.9667132Z" finish="2026-05-16T14:51:28.3117671Z" />
  <TestSettings name="default" id="098aeb3d-18dd-4b7e-bebd-3d6d6cc019de">
    <Deployment runDeploymentRoot="_b96b181dc000000_2026-05-16_14_51_28.3108826" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="192c918b-a046-4527-a919-ecfb7a7b6d1b" testId="18f2dd82-fede-8c47-821a-9500badc6846" testName="FailingTest" computerName="b96b181dc000000" duration="00:00:00.0457987" startTime="2026-05-16T14:51:28.1256581+00:00" endTime="2026-05-16T14:51:28.2268957+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="192c918b-a046-4527-a919-ecfb7a7b6d1b">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 298
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 142
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 48
   at MSTestTrxReport.UnitTest1.FailingTest() in /mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/C9q2X/MSTestTrxReport/UnitTest1.cs:line 11
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="/mnt/vss/_work/1/s/artifacts/tmp/debug/testsuite/c9q2x/mstesttrxreport/bin/release/net10.0/mstesttrxreport.dll" id="18f2dd82-fede-8c47-821a-9500badc6846">
      <Execution id="192c918b-a046-4527-a919-ecfb7a7b6d1b" />
      <TestMethod codeBase="/mnt/vss/_work/1/s/artifacts/tmp/Debug/testsuite/C9q2X/MSTestTrxReport/bin/Release/net10.0/MSTestTrxReport.dll" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="18f2dd82-fede-8c47-821a-9500badc6846" executionId="192c918b-a046-4527-a919-ecfb7a7b6d1b" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="b96b181dc000000" outcome="Error" timestamp="2026-05-16T14:51:28.3298436">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>.

Check failure on line 7199 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Linux Debug)

Build log #L7199 

Bash exited with code '2'.
View more details on Azure Pipelines