Scott Porad

In an automated test environment, it’s important to understand how much code is covered by the automated tests.  Knowing the portion of code that is covered influences trust in the automated testing, and how much manual testing is needed.

There are tools that will do this for you called “code coverage” tools.  (For example, at Cheezburger we use a product called NCover.)

Typically—and forgive me for the gross oversimplification—these tools watch your code while the automation is running and keep track of which lines were executed and which weren’t. The executed lines are considered “covered” and the others are considered “uncovered”.  As a result, what you end up with is a code coverage ratio:

code coverage ratio = covered code / total code

Unfortunately, using a code coverage ratio is not the best way to measure test coverage effectively, and many teams make this mistake.

To understand why, back up and think about the goal of measuring test automation code coverage.  The goal of test coverage is to ensure that all your code is covered.  In other words, that you have zero uncovered code.

When measuring coverage by percentage, it’s easy for the code coverage ratio to look like you’re moving in the right direction when you’re really not.  In other words, yesterday coverage was x%, and today bunch of code was added, and coverage is still x% (or greater than x%), so we’re all good.

But, you’re not…let me illustrate by example: say for example on Monday you have 50% code coverage.  Then, on Tuesday you write a new feature which adds a flim-flam to your wiga-magig.  After you introduce the new feature, you run the coverage report and see that your new code coverage is 51%.  Hooray…the code coverage is moving in the right direction!  That’s good, right?

It is good, but it’s also deceiving.  Let’s look more closely at our codebase:

   Day  LOC*  Covered   Uncovered   Ratio
   Mon  1000      500         500     50%
   Tue  1100      560         540     51%
   * - LOC = Lines of Code

On Tuesday, when you wrote your new feature, there were 100 new lines of code introduced, a 10% increase in the code base.  And, most of those lines of code were covered by automated tests.  But, there are still 40 lines that are uncovered.  In other words, even though your code coverage ratio increased, you moved further away from the goal of zero uncovered lines.

So, what’s the right way to do this?  If the goal is to have zero lines of uncovered code, then the correct metric to monitor is…the most effective way to measure test automation coverage…here it comes…wait for it…lines of uncovered code!

Notes:

• For the purposes of illustration, I used lines of code.  There are more precise metrics than LOC, such as methods or, my preferred, symbol points.
• Also, I do know that coverage isn’t everything.  When thinking about test automation there is “quantity” and “quality”.  Coverage tells you quantity—how much of the code is covered?  It does not tell you quality—are the automated tests any good?