2010/2011 Roy Osherove wrote in his blog a sentence which summarizes an opinion you can still find in many other blogs: „MSTest is the IE6 of Unit Test Frameworks.“ . He basically says, that MS Test has a lack of functionality when it comes to unit testing and I agree with him with one exception:  I think it is not primarily a unit testing framework, it is a test automation framework but lets start with a comparison.

Disadvantages of MS Test 2010

If you compare XUnit.Net, NUnit and MS Test on the surface you will see that they basically do the same. You can define classes and categorize tests which contain test methods and you can validate values with Assert classes. Nobody would bother if this would be every thing we need but with a closer look you see the small differences which lead to a sentence like the one in the title.

With MS Test & Visual Studio 2010 I have following pain points when it comes to unit testing:

  1. Tests are slow compared to other frameworks and you can’t use your studio while it tests. I heard that there shall be a way to switch this off but I never have found it. This is even worse because I work on a solution with around 1500 tests and 50 projects. 
  2. There are 4 different windows where I can get information from or change settings regarding to my tests but all I need is an overview of my tests and ways to define their behavior inside their code.
  3. The .vsmdi files are annoying. You can change settings for your tests outside the code so nobody who works on this code will see it. I had a lot of confusing situations were some of the tests did not run and I had no idea why. These files also tend to increase in unexpected ways when you use TFS because the Studio creates a *.vsmdi file automatically if it can’t save it, and guess what, TFS uses the read-only flag to mark files which are not checked out…
  4. The DeploymentItem is a real mess. It is used to copy files (other than *.dll) and can be defined for each test method but will only deploy the given file once for all. It does not care if the file was changed or deleted. It does not even raise an error if it can’t do it’s work and it forces the framework to copy all the output into a new folder.
  5. I have to use files (and thus the DeploymentItem) and the TestContext when ever I need a parameterized test or row test. I am not able to just define some parameters in my method like it is possible with NUnit or XUnit.Net.
  6. The test runner gets confused if you use base classes for your tests which are not in the same project like your tests. I would like to use this to organize complex test setups for integration and system tests like described in this post but with a bit more voodoo. At least that might not be a big issue for most of the people but I feel limited.
  7. All attributes are sealed. There is no clean way to extend the framework. With clean I don’t mean to change necessarily the registry…

The last two points are real problems for me because I would like to centralize basic setups and test data generation. I have to use helper classes and static factory methods but would like to have something more elegant like the actions in NUnit or the IUseFixture or BeforeAfterTestAttribute of XUnit.Net. How far this can go shows Mark Seemann in a great screen cast. This is not possible and I also have no way to implement it by my self.

Changes in Visual Studio 2012

All these points are valid for VS2010 but hey, the earth is still turning and so a lot of things have changed in Visual Studio 2012 and I am glad about it:

  1. If you are just doing unit testing, than the tests are quite fast thus my main pain point is gone. To keep it fast you should not add Load or Performance tests as well as a *.testsettings file to your test project…
  2. The TestExplorer is not as powerful as the Resharper seems to be but it is much better than all these windows before. I have one window for nearly every thing and see exactly what I need to know. It was also improved with the last two updates for Visual Studio and has now some nice features.
  3. God bless the .vsmdi files are gone.
  4. I can use the DeploymentItem but I don’t have to…
  5. Parameterized tests are still an issue for .Net and I hope this will change in future releases. I think the lack of this feature is a pain point for a lot of the users. That might be the reason why it is available for WinRT since Update 1…
  6. I am not sure about the status of the inheritance of test classes. I don’t expect a change here because it is actually only a problem for a small group of people.
  7. Every thing is still sealed. So what else to say?

Unit vs. Test automation Framework

Like I said at the beginning, I don’t see MS Test as a unit testing framework, mainly because of the lack in speed (only VS2010), extensibility and parameterized tests. On the other hand it is a test automation framework with which you can automated lots of different tests on all levels (system, integration, unit). This point counts more for VS2010 than VS2012 because in 2010 you could generate tests out of production code or use generated accessor classes to test class internals. It might not be the best tool for people who do TDD but it is great if you have a large project where developers and testers work hand in hand or you have a code base which grew over years without any IoC.

You still have for example the Coded UI Tests. The testers can record the actions and the developers combine them in test methods (if the testers don’t want to see code 😉 ). I also like features like the PrivateObject or PrivateType, with which you can easily break dependencies inside applications which does not use IoC. Fakes is another big deal even I still prefer Moq in most of the cases because it handling is often easier.

Conclusion

Roy wrote his post when VS2010 was released and I can understand his point. I often enough thought and think the same about unit testing with VS2010 but if you use VS2012 a lot of the problems are fixed and, if this is not enough, you can easily switch to another framework…

Besides this, I also have to mention that even TDD and unit testing are best practices for developers, they are not the only way to automate tests. So if you go another path you might have other requirements regarding your tools. For example if you do load and performance tests or UI and data base tests.


Kick It auf dotnet-kicks.de