MSTest, also known as Microsoft Testing Framework, simplifies the testing experience for .NET applications. This test framework allows users to write and execute tests while providing test suites integrated into the Test Explorer’s of Visual Studio and Visual Studio Code, as well as many CI pipelines. Fully supported, open-source, and cross-platform, MSTest works with all supported .NET targets while maintaining support for VSTest as well as adding support for the improved experience with Microsoft.Testing.Platform (MTP).
MSTest 3.8 highlights
We continually work to improve MSTest, address your feedback and offer smoother testing experience. MSTest 3.8 is a big release, with lots of great new features:
Filtering: The VSTestBridge extension of Microsoft.Testing.Platform (MTP) 1.6 and later now supports filtering while discovering tests (–list-tests). This feature has been highly requested by users to help verify the tests applicable to a given filter.
Running tests with MSBuild: Extended support for the -t:Test or /t:Test target to support MSBuild.exe.
Improved iterating experience: Updated the Microsoft.Testing.Extensions.TrxReport extension behavior to overwrite the content instead of failing when the TRX file already exists.
Enriched metapackage: Simplified using TRX report and Code Coverage by removing the requirement to manually declare these MTP extensions.
Support of modern UWP: MSTest now supports .NET 9 in UWP.
Improved assertions: Introduced new assertion APIs like Assert.Throws, Assert.ThrowsAsync, Assert.ThrowsExactly, and Assert.ThrowsExactlyAsync.
Better data-driven tests: Parameterized generic test methods are now supported.
Retrying flaky tests: Introduced RetryAttribute to automatically retry failing test methods.
Introducing conditional tests: Added OSConditionAttribute for fine-grained control over where your tests run.
Analyzers: The MSTestAnalysisMode feature allows you to control the severity levels of code analyzers within your test projects.
Read on for more, and check out the full changelog on GitHub for details.
Filtering
The VSTestBridge extension of Microsoft.Testing.Platform (MTP) 1.6 and later now supports filtering while discovering tests (--list-tests
). This feature has been highly requested by users to help verify the tests applicable to a given filter. This gives more confidence that you have the correct filter when you run tests.
Because MSTest with MTP uses the VSTestBridge, that feature is available in MSTest 3.8 and later.
Based on your requests, we also added support for the runsettings TestCaseFilter
element in MTP. Note that this is also available for any test framework relying on Microsoft.Testing.Extensions.VSTestBridge
extension.
Running tests with MSBuild
Another of your requests was to extend the support of the -t:Test
or /t:Test
target that was a dotnet build
only feature to support MSBuild.exe. This is now available for MSTest 3.8, and any test framework based on MTP 1.6+.
Improved iterating experience
We updated the Microsoft.Testing.Extensions.TrxReport
extension behavior when the TRX file already exists to overwrite the content instead of failing. The extension will display a warning message to notify that the file already exists and will be overwritten.
For example, when running the following command twice:
dotnet run -- --report-trx --report-trx-filename test.trx
The second run will not fail, but instead show the following warning:
Warning: Trx file 'path/to/TestResults/test.trx' already exists and will be overwritten.
Enriched metapackage
Since most MSTest projects use both TRX report and Code Coverage, we’ve simplified using them by removing the requirement to manually declare these MTP extensions. With MSTest 3.8 and MTP, you can simplify your projects from:
<ItemGroup>
<PackageReference Include="MSTest" />
<PackageReference Include="Microsoft.Testing.Extensions.CodeCoverage" />
<PackageReference Include="Microsoft.Testing.Extensions.TrxReport" />
</ItemGroup>
to:
<ItemGroup>
<PackageReference Include="MSTest" />
</ItemGroup>
Support of modern UWP
UWP recently announced support for .NET 9 in Modernize your UWP app with preview UWP support for .NET 9 and Native AOT blog post and through close collaboration, we are happy to announce that MSTest supports this new mode. For that to work, you need to be using Microsoft.NET.Test.Sdk
version 17.14.0-preview-25107-01
or above. Stay tuned for a dedicated blog post with all the information and examples coming soon.
Improved assertions
Exception asserts
Historically, only Assert.ThrowsException
and Assert.ThrowsExceptionAsync
were available, requiring an exact match between the specified exception type and the one thrown. Many users requested a simpler naming convention and support for derived exceptions. In response, this release introduces:
Assert.Throws
Assert.ThrowsAsync
Assert.ThrowsExactly
Assert.ThrowsExactlyAsync
To facilitate a smooth upgrade to MSTest 3.8 without causing disruptions to your projects, the old APIs remain available. However, when creating new tests, they will not appear in IntelliSense, encouraging the use of simpler names.
The following is an example of usage of the new APIs:
[TestClass]
public sealed class Test1
{
[TestMethod]
public void TestMethod1()
{
// Will pass because the exceptions have the exact same type
Assert.Throws<OperationCanceledException>(() => throw new OperationCanceledException());
// Will pass because TaskCanceledException derives from OperationCanceledException
Assert.Throws<OperationCanceledException>(() => throw new TaskCanceledException());
// Will fail because the exceptions do not have exactly the same type
Assert.ThrowsExactly<OperationCanceledException>(() => throw new TaskCanceledException());
}
}
We also include MSTEST0039 analyzer along with a codefix to help you with transitioning away from the old APIs to the new ones.
In addition, we added new overloads to the new APIs that accepts a messageBuilder
parameter, which is a Func<Exception?, string>
. This allows you to customize the failure message based on information from the exception thrown (if any). For example:
[TestMethod]
public void TestMethod1()
{
Assert.ThrowsExactly<InvalidOperationException>(
() => throw new ArgumentException("Message of ArgumentException"),
messageBuilder: actualException =>
{
if (actualException is null)
{
return "Expected InvalidOperationException but no exception was thrown.";
}
return $"Expected InvalidOperationException, but found {actualException.GetType().Name} with message {actualException.Message}";
});
}
The above test will fail as follows:
Assert.ThrowsExactly failed. Expected exception type:<System.InvalidOperationException>. Actual exception type:<System.ArgumentException>. Expected InvalidOperationException, but found ArgumentException with message Message of ArgumentException
Collection asserts
We introduced new collection-related assertion APIs in the Assert
class, designed to work with IEnumerable<T>
, expanding the assertion capabilities for collections:
Assert.HasCount
: Asserts that anIEnumerable<T>
has a given number of elements.Assert.IsEmpty
: Asserts that anIEnumerable<T>
doesn’t have any elements.Assert.IsNotEmpty
: Asserts that anIEnumerable<T>
has at least one element.Assert.ContainsSingle
: Asserts that anIEnumerable<T>
has exactly a single element and returns that element.
These new APIs have been added to the Assert
class instead of CollectionAssert
to improve discoverability. We also plan to make all CollectionAssert
APIs available on Assert
.
Interpolated string handler overloads for message parameter
Finally, we added new interpolated string handler overloads for many existing assertion APIs. If you heavily use the Assert
methods overload with message
parameter and use an interpolated string as argument to the message parameter, this will unnecessarily allocate a string that will end up being unused in most cases, as this message is only used in case the assertion failed. Starting with 3.8, new overloads that use interpolated string handler will ensure that the string isn’t allocated unless the assertion fails.
Better data-driven tests
Parameterized generic test methods are now supported, allowing you to write tests like:
[TestClass]
public sealed class TestClass1
{
[TestMethod]
[DataRow(2, 5, 10)]
[DataRow(2.0d, 5.0d, 10.0d)]
[DataRow(5, 5, 25)]
[DataRow(5.0d, 5.0d, 25.0d)]
public void TestMethod1<T>(T number1, T number2, T expectedResult) where T : IMultiplyOperators<T, T, T>
{
Assert.AreEqual(expectedResult, number1 * number2);
}
}
Additionally, we’ve enhanced the behavior of DynamicDataAttribute
to automatically detect the type of the referenced member used as the data source as the default. With this improvement, you can simplify your tests from:
[DynamicData(nameof(MyProperty), DynamicDataSourceType.Property)]
[DynamicData(nameof(MyMethod), DynamicDataSourceType.Method)]
to:
[DynamicData(nameof(MyProperty))]
[DynamicData(nameof(MyMethod))]
Another highly requested feature was the ability to ignore specific data sources from parameterized tests. Starting in 3.8, DataRowAttribute
and DynamicDataAttribute
have an IgnoreMessage
property. When it’s set to non-null string, the source will be ignored. We are also allowing any custom ITestDataSource
to have a similar behavior by simply implementing the ITestDataSourceIgnoreCapability
. With this change, you can now do:
[TestMethod]
[DataRow(0)]
[DataRow(1, IgnoreMessage = "This row is ignored")]
[DataRow(2)]
public void TestMethod1(int x)
{
Assert.AreNotEqual(1, x);
}
or
[TestMethod]
[DynamicData(nameof(Data1))]
[DynamicData(nameof(Data2), IgnoreMessage = "This dynamic data source is ignored")]
[DynamicData(nameof(Data3))]
public void TestMethod1(int x)
{
Assert.IsTrue(x < 10);
}
// 0 to 3 - passes
public static IEnumerable<int> Data1 => Enumerable.Range(0, 4);
// 50 to 59 - ignored, and will fail if not ignored
public static IEnumerable<int> Data2 => Enumerable.Range(50, 10);
// 4 to 9 - passes
public static IEnumerable<int> Data3 => Enumerable.Range(4, 6);
Finally, we have introduced TestDataRow<T>
allowing users to wrap their data in a customizable container where you can set the display name, ignore the entry and much more to come. Any ITestDataSource
can now return IEnumerable<TestDataRow<T>>
. For example:
[TestMethod]
[DynamicData(nameof(GetData))]
public void TestMethod1(int num1, int num2, int expectedSum)
{
Assert.AreEqual(expectedSum, num1 + num2);
}
private static IEnumerable<TestDataRow<(int Num1, int Num2, int ExpectedSum)>> GetData()
{
yield return new((1, 2, 3));
yield return new((2, 3, 5));
yield return new((3, 4, 7)) { DisplayName = "This test case has a special display name" };
yield return new((3, 3, 7)) { IgnoreMessage = "Ignored because of ..." };
}
Retrying flaky tests
We’re excited to introduce RetryAttribute, a powerful new feature that lets you automatically retry failing test methods! With just one attribute, you can configure MaxRetryAttempts
, MillisecondsDelayBetweenRetries
, and BackoffType
(choosing between Constant
and Exponential
strategies for smarter retries). For example:
[TestMethod]
[Retry(3, MillisecondsDelayBetweenRetries = 100, BackoffType = DelayBackoffType.Constant)]
public void TestMethod()
{
Assert.Fail("Failing test");
}
This test method will run four times (the original run and 3 retries), with a time delay of 100 milliseconds between each run.
Need more flexibility? You can create your own custom retry logic by inheriting from RetryBaseAttribute
, the same foundation as the RetryAttribute
.
If you are using MTP, enabling retries across your entire test suite is even easier, just leverage the retry extension and let your tests recover automatically!
Introducing conditional tests
MSTest now includes OSConditionAttribute
, giving you fine-grained control over where your tests run. This attribute lets you specify which operating systems a test should or should not execute on, making it easier to handle OS-specific behavior in your test suite.
Here are some examples of this attribute:
[TestClass]
public sealed class MyTestClass
{
[TestMethod]
[OSCondition(OperatingSystems.Windows)]
public void TestMethodRunningOnlyOnWindows()
{
}
[TestMethod]
[OSCondition(OperatingSystems.Linux | OperatingSystems.OSX)]
public void TestMethodRunningOnlyOnLinuxAndMac()
{
}
[OSCondition(ConditionMode.Exclude, OperatingSystems.Windows)]
public void TestMethodRunningOnAllOSesButWindows()
{
}
}
As part of the development of this feature, we are providing the ConditionBaseAttribute
abstract class as parent of OSConditionAttribute
and IgnoreAttribute
. You can use this new attribute as the root of any conditional feature you would like to implement. We have many ideas for extra conditional features we could provide, and we would really appreciate your input in prioritizing the ones that would be the most useful to you!
Analyzers
The MSTestAnalysisMode
feature allows you to control the severity levels of code analyzers within your test projects. By setting the MSTestAnalysisMode
MSBuild property, you can determine which analyzers are enabled and their corresponding severity levels.
Available Modes:
None
: Disables all analyzers. You can enable individual analyzers using.editorconfig
or.globalconfig
files.Default
: Follows the default behavior for each rule. Rules enabled by default will use their default severity, while those disabled by default will have a severity of none.Recommended
: Elevates rules enabled by default with an Info (suggestion) severity to warnings. Certain rules may be escalated to errors in bothRecommended
andAll
modes.For example,MSTEST0003: Test methods should have valid layout
is escalated to an error in these modes.All
: More aggressive thanRecommended
. All rules are enabled as warnings, with certain rules escalating to errors.
Usage Example:
To set the MSTestAnalysisMode
to Recommended
, include the following in your project file or in Directory.Build.props:
<PropertyGroup>
<MSTestAnalysisMode>Recommended</MSTestAnalysisMode>
</PropertyGroup>
This configuration ensures that your test projects adhere to best practices by appropriately adjusting the severity levels of various analyzers.
Note that using Directory.Build.props is more recommended than project file, as it ensures the property applies to all your test projects and you add it in one place instead of multiple project files. It’s also safe if MSTestAnalysisMode
is defined for non-test projects via Directory.Build.props. It only affects MSTest projects.
For more detailed information, refer to the MSTest code analysis documentation.
Let’s now focus on 2 newly added analyzers that have shown promising results during internal dogfood:
- MSTEST0038: Don’t use ‘Assert.AreSame’ or ‘Assert.AreNotSame’ with value types
- MSTEST0040: Do not assert inside ‘async void’ contexts
MSTEST0038 warns when the methods Assert.AreSame
and Assert.AreNotSame
are called with a value type argument. These methods work by comparing references, making its usage with value types incorrect as it yields unexpected results.
The following use of Assert.AreSame
will always fail:
Assert.AreSame(0, 0);
This is because we pass them to ReferenceEquals
, which accepts object
. So, each argument will be boxed into a different object, and they will not have equal references.
The same applies to Assert.AreNotSame
, but is more critical.
Assert.AreNotSame(0, 0);
By applying the same logic, the above assert will always pass. This can hide real production bugs as you may not be asserting what you want.
Note that the above examples are for demonstration purposes only and are not real-world examples. A more realistic example can be:
// This will still pass even if the array has a single element.
Assert.AreNotSame(1, myArray.Length);
For more information, see MSTEST0038: Don’t use ‘Assert.AreSame’ or ‘Assert.AreNotSame’ with value types.
MSTEST0040 warns for usage of assertion APIs in async void. Depending on whether you use a custom SynchronizationContext
and whether you use VSTest under .NET Framework, an exception thrown in async void
context can either be swallowed completely or can crash the test host. For more information, see MSTEST0040: Do not assert inside ‘async void’ contexts.
Wrapping up
With MSTest 3.8, we’re continuing our mission to make testing in .NET easier, more powerful, and more enjoyable. This release brings highly requested features, reduces tedious tasks, and enhances the overall developer experience — allowing you to focus on writing great tests instead of wrestling with infrastructure.
We hope these improvements help streamline your workflow and showcase our commitment to listening to your feedback and evolving MSTest to meet your needs. As always, we’d love to hear your thoughts — try out MSTest 3.8 and let us know what you think by opening an issue or a discussion on microsoft/testfx GitHub repository!
A huge shoutout to @SimonCropp for his incredible contributions to MTP and MSTest! His dedication and expertise continue to make a lasting impact, and this release is no exception. We’re grateful for his ongoing support and invaluable efforts in helping shape a better testing experience for the entire .NET community. Thank you, Simon!
The post MSTest 3.8: Top 10 features to supercharge your .NET tests! appeared first on .NET Blog.