http://www.developer.com/

Back to article

Automating Software Testing with Microsoft Hyper-V


November 19, 2008

Making sure your software works as it is designed can be tough work, especially if your application is more complex than the ubiquitous Hello World. This is because software works in the context of the operating system and even other applications, which in turn can interfere with your application. This is often the reason why problems at the customer's site are so hard to replicate.

Even though it is the custom of our profession not to give any warranties that applications would always work, software developers still need to make sure their applications work as planned on generic installations. In this testing, using virtual machines can be a big aid. With real hardware, you often find the need to reinstall the operating system, or hot-swap between hard disks to get yourself back to a clean state. Enter virtualization, and leave all that behind.

Testing in a Virtualized Environment

With virtual machines, re-running your tests is often a matter of returning the virtual machine installation to a clean state. A clean state is, in this case, for example a version of the virtual machine that doesn't have certain configurations done, or that your application hasn't yet been installed on it.

Depending on the virtual machine software you use, your also might be able to take snapshots. With snapshots, it is trivial to return a virtual machine to a previous, known state. For example, both Microsoft's Virtual Server and VMware's products, including Workstation, support this feature.

When you return to a snapshot, the virtual machine's hard disk (and memory, if the virtual machine was running at the point when the snapshot was taken) returns to the state it was when the snapshot was taken. This means that if you have, for example, installed applications or changed the configuration of the virtual system, you can undo all those changes and return to the previous snapshot state. Then, you can restart your tests, being absolutely sure no traces or Registry keys of the old installation are left behind (see Figure 1).



Click here for a larger image.

Figure 1: Snapshots, or checkpoints as they are also called, are very helpful in software testing.

Although all this sounds great, working with virtual machines and taking and restoring snapshots can be manual, repetitive work, and thus stealing time from real testing work. Luckily, new virtual machine software allows automation, and this can be key to more effective testing. This is especially true for server-based applications that do not have a user interface.

For example, you could create an automation script that returns a given virtual machine to a clean state, installs your application on the virtual machine, runs a test script, and then takes the results and stores them for later inspection.

When all this work is automated, you could, for example, run automatic application tests nightly, and on the morning of each day have a report telling whether the application worked as planned. If you would combine unit tests into this procedure, you could have a comprehensive set of tests: both at code level and at the runtime level.

Enter Hyper-V

If you have followed Microsoft's latest server features, you might have heard that Windows Server 2008 has built-in support for virtualization. Surely, you could download Virtual Server and install it on any Windows machine recent enough, and run your testing there.

However, the virtualization technique in Windows Server 2008, called Hyper-V, is much more advanced than Virtual Server (see Figure 2). For example, Hyper-V supports multiple processors, better management, and the x64 architecture. Thus, if you want to make sure your applications run properly on Windows Server 2008 (you should!), it's a great idea to set up your virtualization with Hyper-V, and run your tests there.



Click here for a larger image.

Figure 2: Hyper-V brings a lot of power to virtualization.

Hyper-V is a technology that supports enterprise-scale virtualization, and it is also possible to automate it in powerful ways. Originally, Hyper-V came only with properly licensed versions of Windows Server 2008, but just recently, Microsoft announced a product called Hyper-V Server 2008, which is a free product to support virtualization, just like VMware's ESXi.

To use Hyper-V, you need modern hardware: Hyper-V requires both a 64-bit (x64) installation and virtualization support (such as IntelVT) from the processor. Thus, from Intel, you would need some of the latest Core processors or a Xeon processor. But, once these requirements are satisfied, Hyper-V is a big improvement over the same company's older Virtual Server.

Hyper-V provides good management tools already built in. The most important tool, called Hyper-V Manager, allows you to construct new virtual machines, change their settings, and connect to them to view their screens (see Figure 3).



Click here for a larger image.

Figure 3: The Hyper-V Manager utility.

Although the Hyper-V Manager is a nice management tool especially for simpler usage scenarios, it unfortunately does not support automation through scripts or code. For that, you would need a separate management tool, called the System Center Virtual Machine Manager, or System Center VMM. Although this product sounds expensive, it luckily is not. The previous version Virtual Machine Manager cost around $500, and the same could be expected from its successor, the newer 2008 version. At this writing, the 2008 version is still in the beta stage.

With Virtual Machine Manager in place, you will get access to a set of PowerShell commands that let you automate all operations that the graphical management console of Virtual Machine Manager can do. In fact, when you operate VMM's management console thru wizards, on the last page you have to option to show the PowerShell script that would complete the exact same task as the wizard.

A Testing Procedure

Say that you have a.NET application you would like to test as automatically as possible. Armed with a Hyper-V environment and System Center Virtual Machine Manager installation in place, you can write a PowerShell script that fully automates your testing.

For instance, say that your application is a server-side application that talks to a database, does certain calculations and data manipulations, and then writes the results to a log file. If you wanted to test this application manually, you would copy it to a (virtual) testing machine, run the application, and then check the results once they are available.

If the application takes a long time to run—for example, several hours—you might want to refine the testing process even further. For instance, you could write code to send an email saying that your application has finished executing.

Given this flow of logic, your completed testing procedure could have the following steps:

  1. Start a testing virtual machine, and return it to a known, clean state.
  2. Copy and/or install the application to the virtual machine.
  3. Run the application and wait for the results.
  4. Take the log file and copy it to a safe place for later analysis.
  5. Send a notification to the tester(s) telling that there are results to analyze.

Note that although this article talks from the perspective of .NET and Visual Studio development, you actually could use Hyper-V–based testing automation for applications developed with any Windows development tool, be it Embarcadero/CodeGear, Java, PHP, or anything else.

Similarly, you could even test web applications, provided that you have a proper client to access the application automatically. For instance, Visual Studio Team System has support for web tests, which could be exactly what you need.

Next, you are going to learn how to write PowerShell commands to automate the above five-step testing process in a Hyper-V environment (see Figure 4).



Click here for a larger image.

Figure 4: SCVMM's PowerShell interface lets you automate virtual machine operations.

Automating Hyper-V Using PowerShell

When you install System Center Virtual Machine Manager (VMM) management features on a server, you get access to a specially tailored PowerShell command prompt with the proper snap-ins already registered.

VMM supports many different commands thru the PowerShell interface. Just like all other PowerShell commands, VMM's commands are named using a verb-noun pair, such as Get-VM, Restore-Checkpoint, and so on.

Keeping the previous five-step process in mind, you first need to launch the VMM PowerShell console (see Figure 5). This console can be found from the Start menu on the computer on which VMM has been installed. Although you could also fire up a regular PowerShell prompt, the VMM's own shortcut is convenient because it does not require you to load the VMM snap-in module manually.



Click here for a larger image.

Figure 5: Starting the PowerShell interface.

Although there is not yet much documentation available from the VMM's PowerShell commands ("cmdlets"), you can find a list of commands containing, for example, the letters "VM" by typing at the PowerShell prompt:

help *vm*

This would produce a list of commands the letters "vm" anywhere in the command name. Also, typing "help command-name" shows additional help text for the command. This is, at this point, the best way to learn about VMM's PowerShell commands.

To access a certain virtual machine using the PowerShell interface, you need to get an instance of the correct virtual machine object. Because you want to return the correct virtual machine to a correct state, you would restore it to a certain snapshot, or in System Center's parlance, a checkpoint (see Figure 6).



Click here for a larger image.

Figure 6: The Virtual Machine Manager console lets you manage your virtual machines and their checkpoints.

Here are the code lines needed to do this. Note that for simplicity, the code assumes that you have installed the VMM server and the console on the same machine.

$VM = Get-VM -VMMServer hvserver -Name "Windows XP Test"
$restore_to_checkpoint = Get-VMCheckpoint -vm $vm |
   where { $_.Name -eq "Testing Base" }
Restore-VMCheckpoint -VMCheckpoint $restore_to_checkpoint

Here, the first code line connects the Hyper-V server, called "hvserver", and looks for a virtual machine named "Windows XP Test". The resulting object is stored in the variable $VM (in PowerShell, variable names start with the letter $).

The second line uses the previously created virtual machine object $VM to find a checkpoint (snapshot) with the name "Testing Base". This is done with the Get-VMCheckpoint cmdlet, which returns a list of all checkpoints available for the given virtual machine. The "where" clause then finds the correct checkpoint based on the checkpoint's Name property.

The final step is to restore the virtual machine to the given checkpoint with the Restore-VMCheckpoint command. Assuming the virtual machine was running when the checkpoint was taken, the virtual machine will then continue to run from exactly that same point forward.

Filling In the Missing Steps

With the preceding PowerShell commands, you are able to complete the first step of the testing procedure outlined earlier. The following steps are even easier to complete, and they can be done basically with regular DOS commands.

The only obstacle is the capability to execute a process (or run a program) remotely on the virtual machine. Starting a command on a remote machine cannot be done with those old DOS-based commands, but you can use several different options to solve the problem. One is to use a WMI script; the other is to use some simple utility, such as PSExec from Technet SysInternals. The following code uses the free PSExec utility, which is very convenient to use.

Assuming the virtual machine responds in the IP address 192.168.0.123, and would have an accessible network share named "TestLocation", you could use the following PowerShell commands to copy, install, and run the application on the virtual machine:

copy "C:\Source Code\MyApplication\Setup.msi" \\192.168.0.123\
   TestLocation
& 'C:\Program Files (x86)\psexec.exe' \\192.168.0.123 msiexec
   /i /passive C:\TestLocation\Setup.msi
& 'C:\Program Files (x86)\psexec.exe'
   \\192.168.0.123 C:\TestLocation\MyApplication.exe

The first copy command copies the application's setup file to the remote virtual machine, and the second line starts the msiexec utility (the Windows Installer utility) to install the MSI package silently. This is done with the help of the PSExec utility (which is a 32-bit application, so it is located in the x86 version of the Program Files folder). Of course, if your application does not require installation, you could simply replace these lines with an appropriate copy command.

The third line again uses PSExec to remotely start the tested application on the virtual machine. Here, you also could specify whatever parameters might be needed for the application to automatically take the required actions.

Finally, you could collect the results. For instance, if your application logs to a text file, you could simply copy the log back for the testers to investigate:

$log_name = "T:\TestReports\MyApplication Log " +
   [System.DateTime]::Now.ToShortDateString() + ".log";
copy \\192.168.0.123\TestLocation\MyApplication.log $log_name

With these lines, you take the current date from the System.DateTime class (as you see, you can construct .NET class instances directly from the PowerShell prompt), and then combine it to a full log file path. This way, daily logs will not overwrite older logs.

After all is done, you optionally could compose an SMTP message, write an entry to the Windows Event Log, or otherwise signal to the testers that the application has finished running. Alternatively, you could simply schedule the testing to happen at nighttime, and simply view the log file in the morning.

Conclusion

When virtualization in the PC world was new, software developers were one of the first group of users to adopt virtual machine technology. Thus, it is no surprise that virtualization is used heavily in software testing. However, testing can be a time-consuming task if done completely manually.

With Microsoft's Hyper-V virtualization technology, you are able to step up the ladder in virtualization features, and at the same time make your testing more automatic. For example, if you are using Visual Studio Team Foundation Server (TFS) and its Team Build feature, you could automate both the application building and testing in a single MSBuild script, and make all this happen with a click of a button.

Because Hyper-V easily supports multiple virtual machines and dozens of checkpoints (snapshots), you can automate testing of your applications. Automating user interface testing is, of course, more difficult, but even with these applications, being able to automatically configure the virtual testing environment can save a lot of time.

Let these scripts be your starting point in automating your software testing with Microsoft's Hyper-V. Thanks go to Janne Pohjala from Microsoft Finland for helping me find the hardware to test Hyper-V on. With that, happy testing!

Jani Järvinen

Sidebar: Automating Virtual Server and VMware Environments

If you are currently using Microsoft Virtual Server or VMware's Workstation or Server virtualization products and you want to automate your virtual machines with those, you are not out of luck. Both Virtual Server and VMware Workstation offer an automation API that can be used to do similar tasks already shown in this article with Hyper-V.

To get started, check out Microsoft's Virtual Server COM API (callable from C# or any other .NET code) and VMware's VIX API. VMware's API is C based (native, or also known as unmanaged code), but you could also call it from .NET applications.

As for the Virtual Server API, check out the IVMVirtualServer and IVMVirtualMachine interfaces documented on MSDN. For VMware's VIX API, visit www.vmware.com/support/developer/vix-api.

Links

About the Author

Jani Järvinen is a software development trainer and consultant in Finland. He is a Microsoft C# MVP and has written dozens of magazine articles and three books about software development. He is a group leader of a Finnish software development expert group named ITpro.fi. His frequently updated blog can be found at http://www.saunalahti.fi/janij/. You can send him mail by clicking on his name at the top of the article.

Sitemap | Contact Us

Thanks for your registration, follow us on our social networks to keep up-to-date