MSTest 3.8: Top 10 features to supercharge your .NET tests!

MSTest, also known as Microsoft Testing Framework, simplifies the testing experience for .NET applications. This test framework allows users to write and execute tests while providing test suites integrated into the Test Explorer’s of Visual Studio and Visual Studio Code, as well as many CI pipelines. Fully supported, open-source, and cross-platform, MSTest works with all supported .NET targets while maintaining support for VSTest as well as adding support for the improved experience with Microsoft.Testing.Platform (MTP).

MSTest 3.8 highlights

We continually work to improve MSTest, address your feedback and offer smoother testing experience. MSTest 3.8 is a big release, with lots of great new features:

• Filtering: The VSTestBridge extension of Microsoft.Testing.Platform (MTP) 1.6 and later now supports filtering while discovering tests (–list-tests). This feature has been highly requested by users to help verify the tests applicable to a given filter.
• Running tests with MSBuild: Extended support for the -t:Test or /t:Test target to support MSBuild.exe.
• Improved iterating experience: Updated the Microsoft.Testing.Extensions.TrxReport extension behavior to overwrite the content instead of failing when the TRX file already exists.
• Enriched metapackage: Simplified using TRX report and Code Coverage by removing the requirement to manually declare these MTP extensions.
• Support of modern UWP: MSTest now supports .NET 9 in UWP.
• Improved assertions: Introduced new assertion APIs like Assert.Throws, Assert.ThrowsAsync, Assert.ThrowsExactly, and Assert.ThrowsExactlyAsync.
• Better data-driven tests: Parameterized generic test methods are now supported.
• Retrying flaky tests: Introduced RetryAttribute to automatically retry failing test methods.
• Introducing conditional tests: Added OSConditionAttribute for fine-grained control over where your tests run.
• Analyzers: The MSTestAnalysisMode feature allows you to control the severity levels of code analyzers within your test projects.

Read on for more, and check out the full changelog on GitHub for details.

Filtering

The VSTestBridge extension of Microsoft.Testing.Platform (MTP) 1.6 and later now supports filtering while discovering tests (--list-tests). This feature has been highly requested by users to help verify the tests applicable to a given filter. This gives more confidence that you have the correct filter when you run tests.

Because MSTest with MTP uses the VSTestBridge, that feature is available in MSTest 3.8 and later.

Based on your requests, we also added support for the runsettings TestCaseFilter element in MTP. Note that this is also available for any test framework relying on Microsoft.Testing.Extensions.VSTestBridge extension.

Running tests with MSBuild

Another of your requests was to extend the support of the -t:Test or /t:Test target that was a dotnet build only feature to support MSBuild.exe. This is now available for MSTest 3.8, and any test framework based on MTP 1.6+.

Improved iterating experience

We updated the Microsoft.Testing.Extensions.TrxReport extension behavior when the TRX file already exists to overwrite the content instead of failing. The extension will display a warning message to notify that the file already exists and will be overwritten.

For example, when running the following command twice:

dotnet run -- --report-trx --report-trx-filename test.trx

The second run will not fail, but instead show the following warning:

Warning: Trx file 'path/to/TestResults/test.trx' already exists and will be overwritten.

Enriched metapackage

Since most MSTest projects use both TRX report and Code Coverage, we’ve simplified using them by removing the requirement to manually declare these MTP extensions. With MSTest 3.8 and MTP, you can simplify your projects from:

<ItemGroup>
  <PackageReference Include="MSTest" />
  <PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" />
  <PackageReference Include="Microsoft.Testing.Extensions.TrxReport" />
</ItemGroup>

to:

<ItemGroup>
  <PackageReference Include="MSTest" />
</ItemGroup>

Support of modern UWP

UWP recently announced support for .NET 9 in Modernize your UWP app with preview UWP support for .NET 9 and Native AOT blog post and through close collaboration, we are happy to announce that MSTest supports this new mode. For that to work, you need to be using Microsoft.NET.Test.Sdk version 17.14.0-preview-25107-01 or above. Stay tuned for a dedicated blog post with all the information and examples coming soon.

Improved assertions

Exception asserts

Historically, only Assert.ThrowsException and Assert.ThrowsExceptionAsync were available, requiring an exact match between the specified exception type and the one thrown. Many users requested a simpler naming convention and support for derived exceptions. In response, this release introduces:

• Assert.Throws
• Assert.ThrowsAsync
• Assert.ThrowsExactly
• Assert.ThrowsExactlyAsync

To facilitate a smooth upgrade to MSTest 3.8 without causing disruptions to your projects, the old APIs remain available. However, when creating new tests, they will not appear in IntelliSense, encouraging the use of simpler names.

The following is an example of usage of the new APIs:

[TestClass]
public sealed class Test1
{
    [TestMethod]
    public void TestMethod1()
    {
        // Will pass because the exceptions have the exact same type
        Assert.Throws<OperationCanceledException>(() => throw new OperationCanceledException());

        // Will pass because TaskCanceledException derives from OperationCanceledException
        Assert.Throws<OperationCanceledException>(() => throw new TaskCanceledException());  

        // Will fail because the exceptions do not have exactly the same type
        Assert.ThrowsExactly<OperationCanceledException>(() => throw new TaskCanceledException());
    }
}

We also include MSTEST0039 analyzer along with a codefix to help you with transitioning away from the old APIs to the new ones.

In addition, we added new overloads to the new APIs that accepts a messageBuilder parameter, which is a Func<Exception?, string>. This allows you to customize the failure message based on information from the exception thrown (if any). For example:

[TestMethod]
public void TestMethod1()
{
    Assert.ThrowsExactly<InvalidOperationException>(
        () => throw new ArgumentException("Message of ArgumentException"),
        messageBuilder: actualException =>
        {
            if (actualException is null)
            {
                return "Expected InvalidOperationException but no exception was thrown.";
            }

            return $"Expected InvalidOperationException, but found {actualException.GetType().Name} with message {actualException.Message}";
        });
}

The above test will fail as follows:

  Assert.ThrowsExactly failed. Expected exception type:<System.InvalidOperationException>. Actual exception type:<System.ArgumentException>. Expected InvalidOperationException, but found ArgumentException with message Message of ArgumentException

Collection asserts

We introduced new collection-related assertion APIs in the Assert class, designed to work with IEnumerable<T>, expanding the assertion capabilities for collections:

• Assert.HasCount: Asserts that an IEnumerable<T> has a given number of elements.
• Assert.IsEmpty: Asserts that an IEnumerable<T> doesn’t have any elements.
• Assert.IsNotEmpty: Asserts that an IEnumerable<T> has at least one element.
• Assert.ContainsSingle: Asserts that an IEnumerable<T> has exactly a single element and returns that element.

These new APIs have been added to the Assert class instead of CollectionAssert to improve discoverability. We also plan to make all CollectionAssert APIs available on Assert.

Interpolated string handler overloads for message parameter

Finally, we added new interpolated string handler overloads for many existing assertion APIs. If you heavily use the Assert methods overload with message parameter and use an interpolated string as argument to the message parameter, this will unnecessarily allocate a string that will end up being unused in most cases, as this message is only used in case the assertion failed. Starting with 3.8, new overloads that use interpolated string handler will ensure that the string isn’t allocated unless the assertion fails.

Better data-driven tests

Parameterized generic test methods are now supported, allowing you to write tests like:

[TestClass]
public sealed class TestClass1 
{
    [TestMethod]
    [DataRow(2, 5, 10)]
    [DataRow(2.0d, 5.0d, 10.0d)]
    [DataRow(5, 5, 25)]
    [DataRow(5.0d, 5.0d, 25.0d)]
    public void TestMethod1<T>(T number1, T number2, T expectedResult) where T : IMultiplyOperators<T, T, T>
    {
        Assert.AreEqual(expectedResult, number1 * number2);
    }
}

Additionally, we’ve enhanced the behavior of DynamicDataAttribute to automatically detect the type of the referenced member used as the data source as the default. With this improvement, you can simplify your tests from:

[DynamicData(nameof(MyProperty), DynamicDataSourceType.Property)]
[DynamicData(nameof(MyMethod), DynamicDataSourceType.Method)]

to:

[DynamicData(nameof(MyProperty))]
[DynamicData(nameof(MyMethod))]

Another highly requested feature was the ability to ignore specific data sources from parameterized tests. Starting in 3.8, DataRowAttribute and DynamicDataAttribute have an IgnoreMessage property. When it’s set to non-null string, the source will be ignored. We are also allowing any custom ITestDataSource to have a similar behavior by simply implementing the ITestDataSourceIgnoreCapability. With this change, you can now do:

[TestMethod]
[DataRow(0)]
[DataRow(1, IgnoreMessage = "This row is ignored")]
[DataRow(2)]
public void TestMethod1(int x)
{
    Assert.AreNotEqual(1, x);
}

or

[TestMethod]
[DynamicData(nameof(Data1))]
[DynamicData(nameof(Data2), IgnoreMessage = "This dynamic data source is ignored")]
[DynamicData(nameof(Data3))]
public void TestMethod1(int x)
{
    Assert.IsTrue(x < 10);
}

// 0 to 3 - passes
public static IEnumerable<int> Data1 => Enumerable.Range(0, 4);

// 50 to 59 - ignored, and will fail if not ignored
public static IEnumerable<int> Data2 => Enumerable.Range(50, 10);

// 4 to 9 - passes
public static IEnumerable<int> Data3 => Enumerable.Range(4, 6);

Finally, we have introduced TestDataRow<T> allowing users to wrap their data in a customizable container where you can set the display name, ignore the entry and much more to come. Any ITestDataSource can now return IEnumerable<TestDataRow<T>>. For example:

    [TestMethod]
    [DynamicData(nameof(GetData))]
    public void TestMethod1(int num1, int num2, int expectedSum)
    {
        Assert.AreEqual(expectedSum, num1 + num2);
    }

    private static IEnumerable<TestDataRow<(int Num1, int Num2, int ExpectedSum)>> GetData()
    {
        yield return new((1, 2, 3));
        yield return new((2, 3, 5));
        yield return new((3, 4, 7)) { DisplayName = "This test case has a special display name" };
        yield return new((3, 3, 7)) { IgnoreMessage = "Ignored because of ..." };
    }

Retrying flaky tests

We’re excited to introduce RetryAttribute, a powerful new feature that lets you automatically retry failing test methods! With just one attribute, you can configure MaxRetryAttempts, MillisecondsDelayBetweenRetries, and BackoffType (choosing between Constant and Exponential strategies for smarter retries). For example:

[TestMethod]
[Retry(3, MillisecondsDelayBetweenRetries = 100, BackoffType = DelayBackoffType.Constant)]
public void TestMethod()
{
    Assert.Fail("Failing test");
}

This test method will run four times (the original run and 3 retries), with a time delay of 100 milliseconds between each run.

Need more flexibility? You can create your own custom retry logic by inheriting from RetryBaseAttribute, the same foundation as the RetryAttribute.

If you are using MTP, enabling retries across your entire test suite is even easier, just leverage the retry extension and let your tests recover automatically!

Introducing conditional tests

MSTest now includes OSConditionAttribute, giving you fine-grained control over where your tests run. This attribute lets you specify which operating systems a test should or should not execute on, making it easier to handle OS-specific behavior in your test suite.

Here are some examples of this attribute:

[TestClass]
public sealed class MyTestClass
{
    [TestMethod]
    [OSCondition(OperatingSystems.Windows)]
    public void TestMethodRunningOnlyOnWindows()
    {
    }

    [TestMethod]
    [OSCondition(OperatingSystems.Linux | OperatingSystems.OSX)]
    public void TestMethodRunningOnlyOnLinuxAndMac()
    {
    }

    [OSCondition(ConditionMode.Exclude, OperatingSystems.Windows)]
    public void TestMethodRunningOnAllOSesButWindows()
    {
    }
}

As part of the development of this feature, we are providing the ConditionBaseAttribute abstract class as parent of OSConditionAttribute and IgnoreAttribute. You can use this new attribute as the root of any conditional feature you would like to implement. We have many ideas for extra conditional features we could provide, and we would really appreciate your input in prioritizing the ones that would be the most useful to you!

Analyzers

The MSTestAnalysisMode feature allows you to control the severity levels of code analyzers within your test projects. By setting the MSTestAnalysisMode MSBuild property, you can determine which analyzers are enabled and their corresponding severity levels.

Available Modes:

• None: Disables all analyzers. You can enable individual analyzers using .editorconfig or .globalconfig files.
• Default: Follows the default behavior for each rule. Rules enabled by default will use their default severity, while those disabled by default will have a severity of none.
• Recommended: Elevates rules enabled by default with an Info (suggestion) severity to warnings. Certain rules may be escalated to errors in both Recommended and All modes.For example, MSTEST0003: Test methods should have valid layout is escalated to an error in these modes.
• All: More aggressive than Recommended. All rules are enabled as warnings, with certain rules escalating to errors.

Usage Example:

To set the MSTestAnalysisMode to Recommended, include the following in your project file or in Directory.Build.props:

<PropertyGroup>
  <MSTestAnalysisMode>Recommended</MSTestAnalysisMode>
</PropertyGroup>

This configuration ensures that your test projects adhere to best practices by appropriately adjusting the severity levels of various analyzers.

Note that using Directory.Build.props is more recommended than project file, as it ensures the property applies to all your test projects and you add it in one place instead of multiple project files. It’s also safe if MSTestAnalysisMode is defined for non-test projects via Directory.Build.props. It only affects MSTest projects.

For more detailed information, refer to the MSTest code analysis documentation.

Let’s now focus on 2 newly added analyzers that have shown promising results during internal dogfood:

• MSTEST0038: Don’t use ‘Assert.AreSame’ or ‘Assert.AreNotSame’ with value types
• MSTEST0040: Do not assert inside ‘async void’ contexts

MSTEST0038 warns when the methods Assert.AreSame and Assert.AreNotSame are called with a value type argument. These methods work by comparing references, making its usage with value types incorrect as it yields unexpected results.

The following use of Assert.AreSame will always fail:

Assert.AreSame(0, 0);

This is because we pass them to ReferenceEquals, which accepts object. So, each argument will be boxed into a different object, and they will not have equal references.

The same applies to Assert.AreNotSame, but is more critical.

Assert.AreNotSame(0, 0);

By applying the same logic, the above assert will always pass. This can hide real production bugs as you may not be asserting what you want.

Note that the above examples are for demonstration purposes only and are not real-world examples. A more realistic example can be:

// This will still pass even if the array has a single element.
Assert.AreNotSame(1, myArray.Length);

For more information, see MSTEST0038: Don’t use ‘Assert.AreSame’ or ‘Assert.AreNotSame’ with value types.

MSTEST0040 warns for usage of assertion APIs in async void. Depending on whether you use a custom SynchronizationContext and whether you use VSTest under .NET Framework, an exception thrown in async void context can either be swallowed completely or can crash the test host. For more information, see MSTEST0040: Do not assert inside ‘async void’ contexts.

Wrapping up

With MSTest 3.8, we’re continuing our mission to make testing in .NET easier, more powerful, and more enjoyable. This release brings highly requested features, reduces tedious tasks, and enhances the overall developer experience — allowing you to focus on writing great tests instead of wrestling with infrastructure.

We hope these improvements help streamline your workflow and showcase our commitment to listening to your feedback and evolving MSTest to meet your needs. As always, we’d love to hear your thoughts — try out MSTest 3.8 and let us know what you think by opening an issue or a discussion on microsoft/testfx GitHub repository!

A huge shoutout to @SimonCropp for his incredible contributions to MTP and MSTest! His dedication and expertise continue to make a lasting impact, and this release is no exception. We’re grateful for his ongoing support and invaluable efforts in helping shape a better testing experience for the entire .NET community. Thank you, Simon!

The post MSTest 3.8: Top 10 features to supercharge your .NET tests! appeared first on .NET Blog.