Skip to content

Apply structured assertion messages (RFC 012) to AreEqual / AreNotEqual #8231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Evangelink merged 21 commits into from
May 18, 2026

Merge origin/main and resolve conflicts

1950cf2
Select commit
Loading
Failed to load commit list.
Merged

Apply structured assertion messages (RFC 012) to AreEqual / AreNotEqual #8231

Merge origin/main and resolve conflicts
1950cf2
Select commit
Loading
Failed to load commit list.
Azure Pipelines / microsoft.testfx (Build Windows Debug) failed May 16, 2026 in 31m 54s

Build Windows Debug failed

4 errors / 1 warnings

Annotations

Check failure on line 175 in src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Windows Debug)

src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs#L175 

src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs(175,1): error : [ScopeWithSingleFailure_TestFails] [net11.0] Assertion failed. Expected condition to be true.
Expression 'AssertOutputMatchesRegex' failed for member 'ScopeWithSingleFailure_TestFails' at line 30 of file '/_/test/IntegrationTests/MSTest.Acceptance.IntegrationTests/SoftAssertionTests.cs'. Output of the test host is:
Command: D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\6zohP\SoftAssertionTests\bin\Release\net10.0\SoftAssertionTests.exe --filter ScopeWithSingleFailure
====================
ExitCode: 2
====================
StandardOutput:
MSTest v4.3.0-ci (UTC 5/16/2026) [win-x64 - .NET 10.0.8]
failed ScopeWithSingleFailure (54ms)
  Assertion failed. Expected values to be equal.
  expected: 1
  actual:   2
  Assert.AreEqual(1, 2)
    at Microsoft.VisualStudio.TestTools.UnitTesting.AssertScope.Dispose() in /_/src/TestFramework/TestFramework/Assertions/AssertScope.cs:72
    at UnitTest1.ScopeWithSingleFailure() in D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\6zohP\SoftAssertionTests\UnitTest1.cs:23
    at UnitTest1.ScopeWithSingleFailure() in D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\6zohP\SoftAssertionTests\UnitTest1.cs:20
    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Test run summary: Failed! - D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\6zohP\SoftAssertionTests\bin\Release\net10.0\SoftAssertionTests.dll (net10.0|x64)
  total: 1
  failed: 1
  succeeded: 0
  skipped: 0
  duration: 473ms
====================
StandardError:


actual: false

Assert.IsTrue(Regex.IsMatch(testHostResult.StandardOutput, pattern))

Check failure on line 454 in src/TestFramework/TestFramework/Assertions/Assert.Contains.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Windows Debug)

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs#L454 

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs(454,1): error : [TrxReport_WhenTestFails_ContainsExceptionInfoInOutput ("net10.0")] [net11.0] Assert.Contains failed. String '<?xml version="1.0" encoding="utf-8"?>
<TestRun id="73daa811-6cca-4cb5-a434-ac778c41a501" name="cloudtest@3edc424bc000002 2026-05-16 15:12:50.1927551" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T15:12:49.8191764Z" queuing="2026-05-16T15:12:49.8191764Z" start="2026-05-16T15:12:49.8191764Z" finish="2026-05-16T15:12:50.1935944Z" />
  <TestSettings name="default" id="18e9ba21-0510-442a-9914-f54294d09a49">
    <Deployment runDeploymentRoot="cloudtest_3edc424bc000002_2026-05-16_15_12_50.1927551" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="49b42d88-06df-4c2a-9f71-6cd2f699a17c" testId="18f2dd82-fede-8c47-821a-9500badc6846" testName="FailingTest" computerName="3edc424bc000002" duration="00:00:00.0487222" startTime="2026-05-16T15:12:50.0102566+00:00" endTime="2026-05-16T15:12:50.1005710+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="49b42d88-06df-4c2a-9f71-6cd2f699a17c">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 298
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 142
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 48
   at MSTestTrxReport.UnitTest1.FailingTest() in D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\AfxFl\MSTestTrxReport\UnitTest1.cs:line 11
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="d:\a\_work\1\s\artifacts\tmp\debug\testsuite\afxfl\mstesttrxreport\bin\release\net10.0\mstesttrxreport.dll" id="18f2dd82-fede-8c47-821a-9500badc6846">
      <Execution id="49b42d88-06df-4c2a-9f71-6cd2f699a17c" />
      <TestMethod codeBase="D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\AfxFl\MSTestTrxReport\bin\Release\net10.0\MSTestTrxReport.dll" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="18f2dd82-fede-8c47-821a-9500badc6846" executionId="49b42d88-06df-4c2a-9f71-6cd2f699a17c" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="3edc424bc000002" outcome="Error" timestamp="2026-05-16T15:12:50.2115611">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>' does not contain string 'Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;.'. 'substring' expression: '"Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;."', 'value' expression: 'trxContent'. <?xml version="1.0" encoding="utf-8"?>
<TestRun id="73daa811-6cca-4cb5-a434-ac778c41a501" name="cloudtest@3edc424bc000002 2026-05-16 15:12:50.1927551" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T15:12:49.8191764Z" queuing="2026-05-16T15:12:49.8191764Z" start="2026-05-16T15:12:49.8191764Z" finish="2026-05-16T15:12:50.1935944Z" />
  <TestSettings name="default" id="18e9ba21-0510-442a-9914-f54294d09a49">
    <Deployment runDeploymentRoot="cloudtest_3edc424bc000002_2026-05-16_15_12_50.1927551" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="49b42d88-06df-4c2a-9f71-6cd2f699a17c" testId="18f2dd82-fede-8c47-821a-9500badc6846" testName="FailingTest" computerName="3edc424bc000002" duration="00:00:00.0487222" startTime="2026-05-16T15:12:50.0102566+00:00" endTime="2026-05-16T15:12:50.1005710+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="49b42d88-06df-4c2a-9f71-6cd2f699a17c">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 298
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 142
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 48
   at MSTestTrxReport.UnitTest1.FailingTest() in D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\AfxFl\MSTestTrxReport\UnitTest1.cs:line 11
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="d:\a\_work\1\s\artifacts\tmp\debug\testsuite\afxfl\mstesttrxreport\bin\release\net10.0\mstesttrxreport.dll" id="18f2dd82-fede-8c47-821a-9500badc6846">
      <Execution id="49b42d88-06df-4c2a-9f71-6cd2f699a17c" />
      <TestMethod codeBase="D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\AfxFl\MSTestTrxReport\bin\Release\net10.0\MSTestTrxReport.dll" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="18f2dd82-fede-8c47-821a-9500badc6846" executionId="49b42d88-06df-4c2a-9f71-6cd2f699a17c" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="3edc424bc000002" outcome="Error" timestamp="2026-05-16T15:12:50.2115611">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>.

Check failure on line 454 in src/TestFramework/TestFramework/Assertions/Assert.Contains.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Windows Debug)

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs#L454 

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs(454,1): error : [TrxReport_WhenTestFails_ContainsExceptionInfoInOutput ("net462")] [net11.0] Assert.Contains failed. String '<?xml version="1.0" encoding="utf-8"?>
<TestRun id="acfa6b14-f8cc-4c80-a72b-09cd84e48cfa" name="cloudtest@3edc424bc000002 2026-05-16 15:12:50.2971496" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T15:12:49.8602159Z" queuing="2026-05-16T15:12:49.8602159Z" start="2026-05-16T15:12:49.8602159Z" finish="2026-05-16T15:12:50.2976568Z" />
  <TestSettings name="default" id="60f206e6-53d8-4751-a7f0-2d97c0d50d5e">
    <Deployment runDeploymentRoot="cloudtest_3edc424bc000002_2026-05-16_15_12_50.2971496" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2" testId="1951bf25-c1d3-8475-b9c1-c4bd9a3c5dc4" testName="FailingTest" computerName="3edc424bc000002" duration="00:00:00.0362818" startTime="2026-05-16T15:12:50.1470084+00:00" endTime="2026-05-16T15:12:50.2359034+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertFailed(StructuredAssertionMessage structuredMessage)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression)
   at MSTestTrxReport.UnitTest1.FailingTest()</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="d:\a\_work\1\s\artifacts\tmp\debug\testsuite\afxfl\mstesttrxreport\bin\release\net462\mstesttrxreport.exe" id="1951bf25-c1d3-8475-b9c1-c4bd9a3c5dc4">
      <Execution id="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2" />
      <TestMethod codeBase="D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\AfxFl\MSTestTrxReport\bin\Release\net462\MSTestTrxReport.exe" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="1951bf25-c1d3-8475-b9c1-c4bd9a3c5dc4" executionId="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="3edc424bc000002" outcome="Error" timestamp="2026-05-16T15:12:50.3177504">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>' does not contain string 'Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;.'. 'substring' expression: '"Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;."', 'value' expression: 'trxContent'. <?xml version="1.0" encoding="utf-8"?>
<TestRun id="acfa6b14-f8cc-4c80-a72b-09cd84e48cfa" name="cloudtest@3edc424bc000002 2026-05-16 15:12:50.2971496" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T15:12:49.8602159Z" queuing="2026-05-16T15:12:49.8602159Z" start="2026-05-16T15:12:49.8602159Z" finish="2026-05-16T15:12:50.2976568Z" />
  <TestSettings name="default" id="60f206e6-53d8-4751-a7f0-2d97c0d50d5e">
    <Deployment runDeploymentRoot="cloudtest_3edc424bc000002_2026-05-16_15_12_50.2971496" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2" testId="1951bf25-c1d3-8475-b9c1-c4bd9a3c5dc4" testName="FailingTest" computerName="3edc424bc000002" duration="00:00:00.0362818" startTime="2026-05-16T15:12:50.1470084+00:00" endTime="2026-05-16T15:12:50.2359034+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertFailed(StructuredAssertionMessage structuredMessage)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression)
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression)
   at MSTestTrxReport.UnitTest1.FailingTest()</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="d:\a\_work\1\s\artifacts\tmp\debug\testsuite\afxfl\mstesttrxreport\bin\release\net462\mstesttrxreport.exe" id="1951bf25-c1d3-8475-b9c1-c4bd9a3c5dc4">
      <Execution id="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2" />
      <TestMethod codeBase="D:\a\_work\1\s\artifacts\tmp\Debug\testsuite\AfxFl\MSTestTrxReport\bin\Release\net462\MSTestTrxReport.exe" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="1951bf25-c1d3-8475-b9c1-c4bd9a3c5dc4" executionId="4e8cfe88-77be-4b28-b5eb-0eb7859db3e2" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="3edc424bc000002" outcome="Error" timestamp="2026-05-16T15:12:50.3177504">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>.

Check failure on line 28968 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build Windows Debug)

Build log #L28968 

Cmd.exe exited with code '2'.
View more details on Azure Pipelines