Code Coverage Tools Introduction
Today, developers are under more and more stress to deliver more code in less time. For most developers, they use tools to help produce reusable code. This is a really good thing, but with all the code being generated today, how much of the code is instrumented for testing? Forget testing, how well do you know your code?
By the way, most experts state that the test-coverage is around 25 percent across the board and this is after the adoption of great frameworks based on the xUnit approach. But again, the bigger question is how well do you, or how well does your team, know the code? How do you learn about code? I believe one of the great under utilized tools in the developer’s arsenal today is Code Coverage.
Before we get into the benefits of Code Coverage, what is Code Coverage? It is a metric that shows how much of the code has been exercised during the execution of the application. This type of tool has been around a long time and in various formats, but a good Code Coverage tool is essential in writing and understanding great code.
Today’s Code Coverage tools use various approaches to give better understanding of the code being monitored. These include statement coverage; simply put, this is if a statement is executed it reports it as such. Meaning if I have a program that is 100 lines long and 30 of them are executed during the monitoring, it would give me 30 percent coverage. The second type is Condition coverage; what part of a conditional statement gets executed. Then there is the 30,000 foot level or High-level blanket coverage; it gives a total top percentage of code executed against the total lines of known code.
This information can be very enlightening. Think about it… you exercise your application with Code Coverage turned on and the results come back as 35 percent coverage. What’s the deal? Did you just run your application with the “happy path” only or did you truly exercise the application? If you did exercise the application, then it appears a great deal of code may not be needed. However, this assumption may be jumping the gun and a little further analysis will be required.
Before we get into some of the things that Code Coverage can point out, we should take a step back and look at where Code Coverage can be used. The great news is that Code Coverage can be used all over the place–from full Q/A testing and full unit testing, to simple developer testing. Each of these areas can benefit from the use of a code coverage tool.
One test that should be run with Code Coverage is in reference to Q/A. Almost every Q/A department I’ve ever been exposed to has a Q/A script for the application getting tested. This script may be automated or run by hand, but all Q/A departments have a repeatable script. It is always interesting running Code Coverage with that Q/A script. Just like the above example of only 35 percent returned, it is always interesting to see the percentage returned after running a Q/A script. It is usually very low, under 60 percent. What does this mean? First it means 40 percent of the application is not getting tested. Remember, usually these scripts are created over time and are used to judge the quality of a product. Normal IT shops don’t ship, or release a product until it passes the Q/A script. So getting information back that less than 60 percent of the code is being exercised can be very eye-opening.
These numbers are in line with a study (Wiegers 2002) that showed only about 50 – 60 percent of the code was actually touched in a testing script. Think about what this could mean from a developer’s standpoint. What if the company could remove 20 percent of the code that is not used? What would be the cost savings or reduction in overhead if that was possible?
Usually, once this information is known, the Q/A department takes a step back and reviews the areas or functions of the application getting exercised because they are always worried that something will slip through the cracks. But, until Code Coverage, it was always hard to ascertain the percentage of the application getting covered. Now, using Code Coverage the task becomes much easier.
Think about how you can focus around the developer and how Code Coverage can be used with Unit Testing. If you ask most developers if they do a good job testing their code, they would tell you yes. However in another study (Boris Beizer in Johnson 1994), the average programmer thought they tested around 95 percent of the code they wrote, but on further inspection the amount of actual code was more like 30 percent. There is a large difference between those two numbers. Remember how the “happy path” may be the focus, in most developer’s minds, if a procedure is supposed to do a summation of two numbers and the program does that, they are done. But, what happens if a letter or character is exposed to the summation process? Again, most developers only test the first part.
Another thing to keep in mind is the concept of conditionals in code. For each conditional another test has to be written to properly test the other side of the condition. Again, if the testing is only focused on one path, then it is most likely “happy path” testing. One additional tool that should be talked about briefly is using a metric for Cyclomatic Complexity. The Cyclomatic Complexity metric will return the total number of paths though a given method, and the number of paths corresponds to the base number of unit tests needed to test the method. This additional metric is an excellent tool for both unit testing and understanding complexity in code.
Another great side-effect of Code Coverage is around learning code. In the following example, (which may ring true to many developers reading this) the boss comes up to you and states that the State Sales Tax calculation is producing odd numbers every so often and that it needs to be fixed immediately. However, the application in question was just inherited by the developer last week. Where is the State Sales Tax calculation and what parts of the program does it effect? Using Code Coverage, the developer can exercise the application focusing on the State Sales Tax and figure out all the parts of the application that deal with the State Sales Tax. Not only does this save many cycles of looking through code, but it also gives an excellent pointer to the number of unit tests to be created to validate the new Sales Tax Calculation, and also helps protect against rework with introduction of regression bugs.
These are just a few areas where Code Coverage can be applied to help build great software. A word of caution, while Code Coverage is a great tool, it does produce numbers; numbers that can be distorted, numbers that can be misinterpreted, numbers that can lead developers, managers, and companies down a bad path. While Code Coverage will tell you exactly what lines of code are executed, or what percentage of coverage has been obtained, the one thing Code Coverage will not do is tell you how good the code is being executed. A developer could have 100 percent coverage, but the code being tested might not work, produce bad results, or just be plain wrong. Always keep that in perspective when using any metric or tool. High coverage rates do not ensure good code.
Once you get the use of Code Coverage down, it can be used for trending. This is very useful for program managers and companies as a whole. For example, say that starting on week three we had 40 percent total coverage with unit test, week four we had 50 percent of coverage with unit tests, and week five we had 30 percent coverage with unit tests… what gives? Well from week three to four it may have been that the amount of code stayed about the same and more unit tests were created, and between weeks four and five a lot of code was added and very few unit tests were added. It may even mean that some unit tests are being removed. This knowledge can be used to help set goals for the team on the level of coverage they want to achieve. It can also help figure out if problems may be occurring before the developers come back and state they have an issue, allowing for adjustments to be made.
Having knowledge is power, especially when that killer bug comes along and you can point to the amount of unit testing in place, and how the killer bug will not occur again because of the new unit test for that bug. In the same breath, having the knowledge that only 50 percent of the code is getting tested means that the other 50 percent is not, and at a minimum, managers can be made aware of the potential issues or unknowns with the application.
So now We’ve looked at the various ways in which code coverage tools work and how they can be used with Unit Testing. This should give a litle insight into the many ways in which a good code coverage tool is essential in writing, understanding and implementing good code.
About the Author
Mike Rozlog is the Sr. Director Products for Embarcadero Technologies. In this role, he is focused on ensuring the developer focused products being created by Embarcadero meet the expectations of developers around the world. Much of his time is dedicated to discussing and explaining the technical and business aspects of Embarcadero’s products and services to analysts and technical audiences worldwide. Mike was formerly with CodeGear, a developer tools group that was acquired by Embarcadero in 2008. Previously, he spent more than eight years working for Borland in a number of positions, including a primary role as Chief Technical Architect. A reputed author, Mike has been published numerous times. His latest collaboration is Mastering JBuilder from John Wiley & Sons, Inc.