Of all the features in Visual Studio Team System (VSTS), none is more controversial than the unit testing features. The community campaign to convince Microsoft to move unit testing from the premium-level VSTS product into a more readily accessible version has been unsuccessful so far, and many are legitimately concerned that the requirement for VSTS will significantly decrease the uptake of the unit testing features. For all the controversy, the actual functionality of the unit testing has not received a lot of attention. This article examines the VSTS Unit Test offerings from a Visual C++ perspective.
C++ has had the CppUnit unit testing framework available for many years, but it seems like unit testing has not made the inroads in the C++ community that it has in .NET and particularly Java. The reasons for the slow uptake have been varied—native C++ lacks the metadata that makes hooking up unit tests easy and intuitive, CppUnit has not been as polished as its managed contemporaries, and the agile mindset that has been the driving force behind unit testing has not been as globally applicable in the C++ world where large, established projects are more prevalent. VSTS unit testing cannot solve the two intrinsic problems with C++ unit testing (cultural issues related to agile programming and lack of metadata in native C++), but it does deliver a nicely polished unit testing framework that is ready to roll with C++.
Thankfully, the few caveats that do exist with VSTS Unit Testing and C++ are fairly obvious:
- To conduct unit tests for native code, a standard native entry point into that code needs to be exposed.
- The unit testing execution engine in VSTS is designed around managed code and managed code attributes. Hence, native code cannot be used to write unit tests.
- Code generation of unit tests is supported only if the production code (i.e., the code that is to be tested) is compiled with the /clr:safe option.
The fact that the set of limitations is so small is quite impressive—the VSTS team has pushed the functionality right up to the limits of what the language and the .NET and Windows platforms support. The full breakdown of the entire unit testing and C++ combinations and their behaviors is available on the MSDN site.
Authoring and Executing Unit Tests
For those familiar with unit testing frameworks like NUnit, the VSTS unit testing functionality will seem quite familiar. A unit test is a managed method that is marked with the
TestMethod attribute. Unit tests must be included in a type with the
TestClass attribute. Unit tests execute some type of logic and, at the end of executing this logic, compare the result achieved with an expected value using conditional logic in the
Microsoft.VisualStudio.TestTools.UnitTesting,Assert class. If the comparison executed through the Assert class is correct, the unit test passes. Otherwise, it fails.
There is obviously a lot of boilerplate code required before the real functionality of the unit tests can be added. That is where the code generation abilities of VSTS come in. The simplest way to add a unit test for a method is to bring up the context menu in the text editor by right-clicking near the method, and selecting Create Unit Tests… (see Figure 1).
Figure 1. Adding a Unit Test to a C++ Project
Note that this feature will be available only if the code that is being unit tested (referred to as production code in the MSDN documentation) is compiled with /clr:safe. For the simple method shown in Figure 1, the unit test generated will look as follows:
[ TestMethod ]
int i; //
TODO: Initialize to an appropriate value
int j; //
TODO: Initialize to an appropriate value
L”CLRLibrary.Class1.Add did not return the expected value.”);
Assert::Inconclusive(L”Verify the correctness of this test method.”);
Notice that the generated test calls Assert::Inconclusive to let the unit-testing framework know that the test shouldn’t be counted as a passed test. The default result for a test in the absence of a call to an Assert class method is a pass, which can be useful if a test is written purely to check that the tested code runs without throwing an exception (if an unhandled exception is thrown, the test will be defined as a failure). Modifying the unit test so that it actually tests the Add method correctly would result in the following code:
[ TestMethod ]
int i = 5;
int j = 6;
int expected = 11;
int actual = CLRLibrary::Class1::Add(i, j);
Assert::AreEqual(expected, actual, L”CLRLibrary.Class1.Add did” +
” not return the expected value.”);
Unit tests can be run directly from within Visual Studio by selecting either Start Selected Test Project With Debugger or Start Selected Test Project Without Debugger.
Adding a test project to a solution will also result in the creation of a configuration file that defines the various configuration properties for running tests. Thankfully, VSTS also ships with a Visual Studio Addin for managing the configuration file. Double-clicking on the file in Visual Studio will bring up the dialog shown in Figure 2.
Figure 2. Configuring Unit Tests
This dialog allows tests to be configured to run on remote machines; it configures code coverage settings; and it allows scripts for setting up and tearing down before and after a test run to be executed.
VSTS has a great deal of flexibility in selecting which tests to run. The simplest way to configure which tests to run is through the Test Manager. Selecting Tests | Windows | Test Manager will bring up the Test Manager, as shown in Figure 3.
Figure 3. Test Manager
It is possible to create a filter that runs tests only with a field that matches a particular keyword. You also can create a custom Test List and drag-and-drop tests into the List. Once a filter or Test List has narrowed the tests that need to be run, the selected tests can be run with or without the debugger. The test management settings built using the Test Manager are stored in a Visual Studio Metadata file, which is an XML file with a vsmdi extension. This file is stored with the solution, and settings made in the Test Manager are not lost when it is closed.
The test configuration file dialog shown in Figure 2 can be used to instrument binary files for code coverage analysis. Once the instrumentation has been added and the tests re-run, the code coverage results can be brought up with a context menu in the Test Results window, as shown in Figure 4.
Figure 4. Accessing Code Coverage Results
Unit Testing C++ Made Easy
For those that have found getting into unit testing with C++ too hard, the release of VSTS represents an excellent opportunity to have another shot at integrating unit testing into your development practices.
About the Author
Nick Wienholt is an independent Windows and .NET consultant based in Sydney, Australia. He is the author of Maximizing .NET Performance from Apress, and specializes in system-level software architecture and development with a particular focus on performance, security, interoperability, and debugging. Nick can be reached at NickW@dotnetperformance.com.