Skip to content

Apply structured assertion messages (RFC 012) to AreEqual / AreNotEqual #8231

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Evangelink wants to merge 19 commits into
base: main
Choose a base branch
from

Merge origin/main and resolve conflicts

1950cf2
Select commit
Loading
Failed to load commit list.
Open

Apply structured assertion messages (RFC 012) to AreEqual / AreNotEqual #8231

Merge origin/main and resolve conflicts
1950cf2
Select commit
Loading
Failed to load commit list.
Azure Pipelines / microsoft.testfx (Build MacOS Debug) failed May 16, 2026 in 35m 8s

Build MacOS Debug failed

4 errors / 0 warnings

Annotations

Check failure on line 175 in src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build MacOS Debug)

src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs#L175 

src/TestFramework/TestFramework/Assertions/Assert.IsTrue.cs(175,1): error : [ScopeWithSingleFailure_TestFails] [net11.0] Assertion failed. Expected condition to be true.
Expression 'AssertOutputMatchesRegex' failed for member 'ScopeWithSingleFailure_TestFails' at line 30 of file '/_/test/IntegrationTests/MSTest.Acceptance.IntegrationTests/SoftAssertionTests.cs'. Output of the test host is:
Command: /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/ChotE/SoftAssertionTests/bin/Release/net10.0/SoftAssertionTests --filter ScopeWithSingleFailure
====================
ExitCode: 2
====================
StandardOutput:
MSTest v4.3.0-ci (UTC 5/16/2026) [osx-x64 - .NET 10.0.8]
failed ScopeWithSingleFailure (765ms)
  Assertion failed. Expected values to be equal.
  expected: 1
  actual:   2
  Assert.AreEqual(1, 2)
    at Microsoft.VisualStudio.TestTools.UnitTesting.AssertScope.Dispose() in /_/src/TestFramework/TestFramework/Assertions/AssertScope.cs:72
    at UnitTest1.ScopeWithSingleFailure() in /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/ChotE/SoftAssertionTests/UnitTest1.cs:23
    at UnitTest1.ScopeWithSingleFailure() in /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/ChotE/SoftAssertionTests/UnitTest1.cs:20
    at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
    at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)
Test run summary: Failed! - /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/ChotE/SoftAssertionTests/bin/Release/net10.0/SoftAssertionTests.dll (net10.0|x64)
  total: 1
  failed: 1
  succeeded: 0
  skipped: 0
  duration: 2s 306ms
====================
StandardError:


actual: false

Assert.IsTrue(Regex.IsMatch(testHostResult.StandardOutput, pattern))

Check failure on line 454 in src/TestFramework/TestFramework/Assertions/Assert.Contains.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build MacOS Debug)

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs#L454 

src/TestFramework/TestFramework/Assertions/Assert.Contains.cs(454,1): error : [TrxReport_WhenTestFails_ContainsExceptionInfoInOutput ("net10.0")] [net11.0] Assert.Contains failed. String '<?xml version="1.0" encoding="utf-8"?>
<TestRun id="5e85878b-8c14-4d66-979b-beaf878a762f" name="@iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941 2026-05-16 15:07:34.2949970" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T15:07:29.993541Z" queuing="2026-05-16T15:07:29.993541Z" start="2026-05-16T15:07:29.993541Z" finish="2026-05-16T15:07:34.296197Z" />
  <TestSettings name="default" id="a77e95e2-520d-4edd-a08b-ac730f558f00">
    <Deployment runDeploymentRoot="_iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941_2026-05-16_15_07_34.2949970" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="d5d6e0dd-dbc9-461b-873c-a9639d831be7" testId="18f2dd82-fede-8c47-821a-9500badc6846" testName="FailingTest" computerName="iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941" duration="00:00:00.0825942" startTime="2026-05-16T15:07:30.4131450+00:00" endTime="2026-05-16T15:07:30.5700520+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="d5d6e0dd-dbc9-461b-873c-a9639d831be7">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 298
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 142
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 48
   at MSTestTrxReport.UnitTest1.FailingTest() in /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/AZX7c/MSTestTrxReport/UnitTest1.cs:line 11
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="/users/runner/work/1/s/artifacts/tmp/debug/testsuite/azx7c/mstesttrxreport/bin/release/net10.0/mstesttrxreport.dll" id="18f2dd82-fede-8c47-821a-9500badc6846">
      <Execution id="d5d6e0dd-dbc9-461b-873c-a9639d831be7" />
      <TestMethod codeBase="/Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/AZX7c/MSTestTrxReport/bin/Release/net10.0/MSTestTrxReport.dll" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="18f2dd82-fede-8c47-821a-9500badc6846" executionId="d5d6e0dd-dbc9-461b-873c-a9639d831be7" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941" outcome="Error" timestamp="2026-05-16T15:07:34.321181">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>' does not contain string 'Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;.'. 'substring' expression: '"Assert.AreEqual failed. Expected:&lt;1&gt;. Actual:&lt;2&gt;."', 'value' expression: 'trxContent'. <?xml version="1.0" encoding="utf-8"?>
<TestRun id="5e85878b-8c14-4d66-979b-beaf878a762f" name="@iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941 2026-05-16 15:07:34.2949970" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <Times creation="2026-05-16T15:07:29.993541Z" queuing="2026-05-16T15:07:29.993541Z" start="2026-05-16T15:07:29.993541Z" finish="2026-05-16T15:07:34.296197Z" />
  <TestSettings name="default" id="a77e95e2-520d-4edd-a08b-ac730f558f00">
    <Deployment runDeploymentRoot="_iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941_2026-05-16_15_07_34.2949970" />
  </TestSettings>
  <Results>
    <UnitTestResult executionId="d5d6e0dd-dbc9-461b-873c-a9639d831be7" testId="18f2dd82-fede-8c47-821a-9500badc6846" testName="FailingTest" computerName="iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941" duration="00:00:00.0825942" startTime="2026-05-16T15:07:30.4131450+00:00" endTime="2026-05-16T15:07:30.5700520+00:00" testType="13CDC9D9-DDB5-4fa4-A97D-D965CCFC6D4B" outcome="Failed" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" relativeResultsDirectory="d5d6e0dd-dbc9-461b-873c-a9639d831be7">
      <Output>
        <ErrorInfo>
          <Message>Assertion failed. Expected values to be equal.

expected: 1
actual:   2

Assert.AreEqual(1, 2)</Message>
          <StackTrace>   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.ReportAssertAreEqualFailed(Object expected, Object actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 298
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, IEqualityComparer`1 comparer, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 142
   at Microsoft.VisualStudio.TestTools.UnitTesting.Assert.AreEqual[T](T expected, T actual, String message, String expectedExpression, String actualExpression) in /_/src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs:line 48
   at MSTestTrxReport.UnitTest1.FailingTest() in /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/AZX7c/MSTestTrxReport/UnitTest1.cs:line 11
   at System.Reflection.MethodBaseInvoker.InterpretedInvoke_Method(Object obj, IntPtr* args)
   at System.Reflection.MethodBaseInvoker.InvokeWithNoArgs(Object obj, BindingFlags invokeAttr)</StackTrace>
        </ErrorInfo>
      </Output>
    </UnitTestResult>
  </Results>
  <TestDefinitions>
    <UnitTest name="FailingTest" storage="/users/runner/work/1/s/artifacts/tmp/debug/testsuite/azx7c/mstesttrxreport/bin/release/net10.0/mstesttrxreport.dll" id="18f2dd82-fede-8c47-821a-9500badc6846">
      <Execution id="d5d6e0dd-dbc9-461b-873c-a9639d831be7" />
      <TestMethod codeBase="/Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/AZX7c/MSTestTrxReport/bin/Release/net10.0/MSTestTrxReport.dll" adapterTypeName="executor://MSTestExtension/4.3.0-ci" className="MSTestTrxReport.UnitTest1" name="FailingTest" />
    </UnitTest>
  </TestDefinitions>
  <TestEntries>
    <TestEntry testId="18f2dd82-fede-8c47-821a-9500badc6846" executionId="d5d6e0dd-dbc9-461b-873c-a9639d831be7" testListId="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
  </TestEntries>
  <TestLists>
    <TestList name="Results Not in a List" id="8C84FA94-04C1-424b-9868-57A2D4851A1D" />
    <TestList name="All Loaded Results" id="19431567-8539-422a-85d7-44ee4e166bda" />
  </TestLists>
  <ResultSummary outcome="Failed">
    <Counters total="1" executed="1" passed="0" failed="1" error="0" timeout="0" aborted="0" inconclusive="0" passedButRunAborted="0" notRunnable="0" notExecuted="0" disconnected="0" warning="0" completed="0" inProgress="0" pending="0" />
    <RunInfos>
      <RunInfo computerName="iad01-dz256-17465d49-9651-46a3-a964-2618e8fcbdc5-8AD4D9D08941" outcome="Error" timestamp="2026-05-16T15:07:34.321181">
        <Text>Exit code indicates failure: '2'. Please refer to https://aka.ms/testingplatform/exitcodes for more information.</Text>
      </RunInfo>
    </RunInfos>
  </ResultSummary>
</TestRun>.

Check failure on line 259 in src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build MacOS Debug)

src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs#L259 

src/TestFramework/TestFramework/Assertions/Assert.AreEqual.cs(259,1): error : [HangDump_TemplateFileName_CreateDump ("net10.0")] [net11.0] Assert.AreEqual failed. Expected:<7>. Actual:<0>. 'expected' expression: 'exitCode', 'actual' expression: 'testHostResult.ExitCode'. Expression 'AssertExitCodeIs' failed for member 'HangDump_TemplateFileName_CreateDump' at line 121 of file '/_/test/IntegrationTests/Microsoft.Testing.Platform.Acceptance.IntegrationTests/HangDumpTests.cs'. Output of the test host is:
Command: /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/1WqnK/AssetFixture/bin/Release/net10.0/HangDump --hangdump --hangdump-timeout 8s --hangdump-filename {pname}_{pid}_{tfm}_{time}_hang.dmp --results-directory /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/1WqnK/AssetFixture/07438228cdd44e8f9471c72fcfb10b55/net10.0
====================
ExitCode: 0
====================
StandardOutput:
Microsoft.Testing.Platform v2.3.0-ci (UTC 05/16/2026) [osx-x64 - net10.0]
Hang dump timeout of '00:00:08' expired
Dumping process 80670 - HangDump
Creating dump file '/Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/1WqnK/AssetFixture/07438228cdd44e8f9471c72fcfb10b55/net10.0/HangDump_80670_net10.0_2026-05-16_14-58-40.9076110_hang.dmp'
[createdump] Gathering state for process 80670 
[createdump] Writing full dump to file /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/1WqnK/AssetFixture/07438228cdd44e8f9471c72fcfb10b55/net10.0/HangDump_80670_net10.0_2026-05-16_14-58-40.9076110_hang.dmp
[createdump] Written 5644349736 bytes (1378015 pages) to core file
[createdump] Target process is alive
Test run summary: Passed! - /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/1WqnK/AssetFixture/bin/Release/net10.0/HangDump.dll (net10.0|x64)
  total: 2
  failed: 0
  succeeded: 2
  skipped: 0
  duration: 2m 53s 011ms
[createdump] Dump successfully written in 159323ms
  Out of process file artifacts produced:
    - /Users/runner/work/1/s/artifacts/tmp/Debug/testsuite/1WqnK/AssetFixture/07438228cdd44e8f9471c72fcfb10b55/net10.0/HangDump_80670_net10.0_2026-05-16_14-58-40.9076110_hang.dmp
====================
StandardError:

Check failure on line 7274 in Build log

See this annotation in the file changed.

@azure-pipelines azure-pipelines / microsoft.testfx (Build MacOS Debug)

Build log #L7274 

Bash exited with code '2'.
View more details on Azure Pipelines