I am a huge proponent of test automation.  That is, having robots that exercise the code to verify that it’s working as expected.  There are numerous and endless reasons for why test automation is important, but I’ll leave it up to you to search the Interwebs and learn why.

In an automated test environment, it’s important to understand how much code is covered by the automated tests.  Knowing the portion of code that is covered influences trust in the automated testing, and how much manual testing is needed.

There are tools that will do this for you called “code coverage” tools.  (For example, at Cheezburger we use a product called NCover.)

Typically—and forgive me for the gross oversimplification—these tools watch your code while the automation is running and keep track of which lines were executed and which weren’t. The executed lines are considered “covered” and the others are considered “uncovered”.  As a result, what you end up with is a code coverage ratio:

code coverage ratio = covered code / total code

Unfortunately, using a code coverage ratio is not the best way to measure test coverage effectively, and many teams make this mistake.

To understand why, back up and think about the goal of measuring test automation code coverage.  The goal of test coverage is to ensure that all your code is covered.  In other words, that you have zero uncovered code.

When measuring coverage by percentage, it’s easy for the code coverage ratio to look like you’re moving in the right direction when you’re really not.  In other words, yesterday coverage was x%, and today bunch of code was added, and coverage is still x% (or greater than x%), so we’re all good.

But, you’re not…let me illustrate by example: say for example on Monday you have 50% code coverage.  Then, on Tuesday you write a new feature which adds a flim-flam to your wiga-magig.  After you introduce the new feature, you run the coverage report and see that your new code coverage is 51%.  Hooray…the code coverage is moving in the right direction!  That’s good, right?

It is good, but it’s also deceiving.  Let’s look more closely at our codebase:

   Day  LOC*  Covered   Uncovered   Ratio
   Mon  1000      500         500     50%
   Tue  1100      560         540     51%
   * - LOC = Lines of Code

On Tuesday, when you wrote your new feature, there were 100 new lines of code introduced, a 10% increase in the code base.  And, most of those lines of code were covered by automated tests.  But, there are still 40 lines that are uncovered.  In other words, even though your code coverage ratio increased, you moved further away from the goal of zero uncovered lines.

So, what’s the right way to do this?  If the goal is to have zero lines of uncovered code, then the correct metric to monitor is…the most effective way to measure test automation coverage…here it comes…wait for it…lines of uncovered code!

Notes:

• For the purposes of illustration, I used lines of code.  There are more precise metrics than LOC, such as methods or, my preferred, symbol points.
• Also, I do know that coverage isn’t everything.  When thinking about test automation there is “quantity” and “quality”.  Coverage tells you quantity—how much of the code is covered?  It does not tell you quality—are the automated tests any good?

There’s an expression I’ve heard that says there are two types of work: working in your business and working on your business.  What’s the difference?

Working in your business is doing the work of making the business go.  For example, if you have a widget factory, you’re working in your business when you’re making widgets.

Working on your business is doing the work of improving the business.  For example, if you have a widget factory, you’re working on your business when you streamline your processes to produce more widgets at a lower cost.

There’s a Japanese work kaizen which literally means “improvement” but has come to reflect a philosophy of continuous and regular working on the business.

Last week, the CheezTech team made an effort at kaizen—we took an hour out of our busy schedules to talk about how we could improve our development processes and productivity.  The results, both practical and emotional, were remarkable.

At the end of our hour we walked away with three concrete changes to our processes.  And, we committed to meet again in a month to see how we’ve done, and look for other improvements.  The entire team felt great!

Today, several members of our team attended a conference and participated in an experiential session that used game-like activities to teach about teamwork and productivity.  In one of the activities, the group was divided in two teams and each team was asked to move as many baseballs as possible from here-to-there following certain rules.

We did it once, and our efficiency was measured.  Then, we were given exactly two minutes to discuss as a team how we could improve our process before we had to do it again.  Just two minutes of working on our business resulted in an almost 100% improvement in efficiency!

Not only did we learn the importance of taking a time-out to discuss improvements as a team, perhaps the most important lesson had to do with the difficulty of implementing changes “to the process” while “in process”.

During the first attempt, two of the 14 members of the team discovered an opportunity to streamline, but were unable to effectively communicate it to the rest of the team.  Since the whole team was busy working in the business, it unable to divert attention and communicate in order to make improvements on-the-fly.

The lesson here is clear: there is genuine value in taking small amounts of time to step outside of working in the business and gather as a team and discuss how improvements can be made to the business.  Taking this time to step aside is critical because it’s difficult to apply changes in-process.

I’m excited for this new aspect of our teamwork at Cheezburger.  By itself, introducing our monthly meeting is an excellent improvement.  I’ll report back you next month to tell you how it goes.

P.S. For those of you familiar with Agile and Scrum development methodologies, this type of kaizen practice is built into the methodology by way of the “sprint retrospective meeting”.

Ed. Note: This is a post from an old development blog I used to write that I don’t maintain any longer.  But, since I’ve been writing about test-driven development lately, I wanted to repost it.  Enjoy!

My friend Mark and I have talked a lot in the past about doing more at unit testing. That’s something I’ve really been focusing on lately and one of the things that has come out of it is that code written for unit tests is subtly different than code not. This became crystal clear to me just a moment ago as I was writing this:

http://pastie.org/322821

The first way I wrote it was a somewhat untestable way of doing things: in order to test AllowPost my unit test had to know all about our configuration stuff and it just couldn’t easily be tested without a bunch of dependencies. Of course, my unit test framework doesn’t know about those dependencies without making it jump through a whole bunch of hoops. By re-writing it the second way, the method AllowPost was an atomic unit that could be tested without knowing about the other stuff.

Of course, I want some tests that cover “the other stuff”, but that’s at a higher level. First, I have to get the lowest levels of tests working before the higher level tests will, otherwise it’s all spaghetti.

This may all be old news to you, but for me it really drove home the phrase “test-driven development”…in other words, the tests are actually shaping the code.  The tests are driving the development.

Suddenly, it makes so much sense why, in order for TDD to be effective, you have to write the tests first. Otherwise, you may end up with code that isn’t testable in terms of a unit test framework.

After yesterday’s post on reaching a milestone on the way to our test automation goals, I thought I would share with you an example on why test automation makes so much sense.

Imagine you have 1 developer and 1 tester.  Each feature takes 1 day to develop and 1 day to test and ship.

• For the first feature, it will take 2 days to ship–1 day to develop and 1 day to test.
• For the second feature, it will take 3 days to ship–1 day to develop, 1 day to test the feature, and 1 day to regression test the first feature.
• For the third feature, it will take 4 days to ship–1 day to develop, 1 day to test the feature, and 2 days to regression test the first and second features.

For the fourth feature…you know how it goes.  By the time you get to 10 features it will take you 11 days to develop, test, regression test and ship your code.

Of course, in the real world that never happens.  In the real world, some relatively arbitrary amount of time is determined for testing based on how many new features are being added to the site.  As a result, all the previous features only get a cursory regression, or they get no regression at all.

In other words, without test automation, the more features you have, the less quality assurance that you do because it becomes prohibitively expensive to ensure complete test coverage.

Now, add test automation: it takes 1 day to develop, 1.5 days to write the tests, and 0.5 days to run tests for all of your features and ship.

• Feature #1: 3 days to ship–1 day develop, 1.5 days write tests, 0.5 days run tests and ship.
• Feature #2: 3 days to ship–1 day develop, 1.5 days write tests, 0.5 days run tests and ship.
• Feature #3: 3 days to ship–1 day develop, 1.5 days write tests, 0.5 days run tests and ship.

For the fourth feature…do you see the difference here?  If you do the math, by the time you ship your 10th feature it will have taken you 30 days.  With manual testing it will have taken 65 days!  That’s right–it will have taken you less than half the time!

Or, what happens in reality is that you will spend 30 days on both approaches, and then your product will only be half as good with manual testing.

Now, I get this is a simple example, but map it out over a few months or a year or the lifetime of your web site and you’ll see what I do: test automation simply makes sense.

This morning I realized that we’re half way to achieving our test automation goals.  Excellent!  I believe that success begets success, so I find it motivating to have achieved this milestone.  I am encouraged to complete the second half and achieve our goal.

We didn’t launch Cheezburger with complete test automation.  It’s a long story as to why, but I’ll make it short: as a startup, sometimes corners get cut in the name of progress.  Unfortunately, cutting test automation is probably one of the most expensive corners to cut.  Simply put, writing code without test automation adds cost in the future.

For one, it’s really time consuming and expensive to go back and write tests for code that has already been written.  If it adds X more effort to a project to write the tests first, then in my experience it adds 3X to write them days, weeks or months later.

In addition, manually tested code is never tested as well as code that is tested by machines.  This is because as the feature set grows, each feature gets less and less human attention on each test pass.  Machine testing gives equal attention on every test pass.  The lower quality testing inevitably leads to more bugs which result in a lower quality product (and takes time to fix).

Many developers complain that writing tests takes longer and they have a lot of pressure to ship code faster.  WRONG!  There are plenty of studies and examples that show greater effectiveness and lower total cost with test automation.  A Microsoft Research study on Test-Driven Development (TDD) shows that initial development took 15-35% longer initially, but reduced bugs 40-90%.  I can guarantee you dollars-to-donuts that a project with a half less bugs ships faster.

I tell my team that writing tests is simply part of the project.  When a developer takes on a project writing the code is only a portion of the job–writing the tests is the other portion.  The developers and teams that understand that will be more successful, I am certain.

Also, I like to tell my team that automated testing will make them happier.  When you ship a project you want to celebrate the achievement, not be dragged down by issues and problems immediately thereafter.  Automated testing–done well–reduces these issues leading to more successful launches.  In my experience, people are happier when they have success.

Bottom line: I’m proud of my team for having reached this milestone, and I’m eager and motivated to continue on and achieve our goal.

It is ridiculously hot in Seattle right now. We’re in the middle of a very unusual heat wave–I’ve lived here for the majority of my life and I don’t ever remember a stretch like this! Most people here aren’t prepared for hot weather–I’d say at most 5% of homes have air conditioning.

But, I’m not complaining. Since it’s so often cloudy and overcast in Seattle, I try very hard not to complain about the times when it’s uncomfortably hot. In fact, I think there’s a lesson to be learned from this heat that can be applied to developing a web site.

All-Time Record High for Seattle

When it comes to the weather, Mother Nature doesn’t always cooperate with what us hooomans, as the LOLcats like to say, have planned. We plan a wedding, but it rains. We plan to work, but it’s 99 degrees and there’s no AC in the office, so it’s impossible to concentrate.

In other words, there are things beyond our control, and accepting that is important. Learning to go with the flow is a very valuable skill.

One thing that’s beyond our control is knowing exactly how long it’s going to take to get a project done.

For example, we’ve been working on a project at Cheezburger that we’re going to be rolling out in phases over the next few months. Our plan was to ship the first phase this week, but it’s not ready yet. At that point, we sort of had some options:

We could be rigid, and have shipped when it wasn’t ready simply for the purpose of “meeting our schedule”. But, we’re just going to spend the next week fixing the things that we’re ready. The customers will be unhappy, the bosses will be aggravated, and the developers will be demoralized. This is a lousy plan.

We could beat ourselves up and burn the midnight oil, so that it “shipped on schedule”. While that may ship this phase on time, and make the dev team feel like heroes, it will likely result in a delay on the next phase because a) we will have burned ourselves out, b) haste makes waste (i.e. bugs) that we’ll have to spend time fixing. This is also a lousy plan.

Or, we could accept reality: it’s taking us between 3-7 days longer than we thought it would. (Rome wasn’t built in a day, as they say. But, I’m also pretty sure that when they started building Rome they couldn’t predict accurately how long it would take.)

Personally, I don’t see a lot of value in trying to bend reality toward arbitrary hooman schedules. Nobody wins with rigidity–it just creates a lot of extra effort and stress without much benefit. Just like we can’t always control or predict the weather, sometimes we can’t control or predict how a project will go either.

You should sign up for my RSS feed which you can do by clicking here. Subscribing to my RSS feed will save you time by pushing my blog directly to you, and ensure that you don’t miss anything important.

This week I wrote about launching and swarming.  It wasn’t by accident that I wrote about both these topics this week–it was deliberate.

In Launching: The Only Thing That Matters, I made the point that your startup couldn’t realize the value they were creating without putting the product in front of customers.  In other words, if you don’t start the race, you can’t win.

Want to Get More Done? Here’s How: Do Less! illustrated how by focusing on one project at at time a team can deliver results more quickly while providing the business greater flexibility and eliminating waste.

Why did I write about them together?  What do they have in common?

The answer is if launching as soon as possible is a key to success, then the “swarm” method of project development is an essential ingredient.  The startups that swarm will launch more quickly and, therefore, have an advantage over those that don’t.  Startups are hard, so every bit of advantage is priceless.

I gave a talk recently about some of the development practices we follow at Cheezburger.  The structure of the talk was a series of questions, one of them being:

Why try to get a lot of stuff done when you could just do one thing instead?

All companies want to get as much done as possible, and most companies try to accomplish this by doing a lot of things.  Our experience has found that doing less things actually results in greater productivity.

We call our approach the “Swarm method”.  I don’t think we invented Swarm, but we’ve found success with it.  In a nutshell, Swarm says that you’re entire team should work on one project at a time until it is complete and has shipped.  Then, like swarm of bees, the team should move to the next project all at once.

What!?  How could this possibly be better than working on many projects at once?  Let me show you how:

• Imagine you have a team with 5 developers, or 25 dev days per week.
• Imagine you have five 1 week projects, or 25 dev days of work.
• In the “Do Lots” method each developer is assigned a project and they’re all done in one week.
• In the Swarm method, the entire team works on a project for a day and they’re all done in a week.

So, I guess there’s not really a difference, right?  Wrong.  What happens on Tuesday when the boss changes his or her mind about the project priorities.

• In the Do Lots method, no projects are ready to ship, so two days have been wasted.
• In the Swarm method, two projects are ready to ship, so a) something got done, and b) no days were wasted.

In short, the moral of the story is:

Being Busy != Being Productive

Furthermore:

Being Productive = Shipping Code

Shipped Code = Opportunity to Create Value

If you’re working in a dynamic, startup environment you cannot afford to waste your time; every moment of your time needs to be focused on creating value for your startup.  And, you cannot afford to be inflexible or you will miss opportunities.  Swarm solves these problems.

You should sign up for my RSS feed which you can do by clicking here. Subscribing to my RSS feed will save you time by pushing my blog directly to you, and ensure that you don’t miss anything important.

I was inspired to write the Launching is the Only Thing That Matters post based on an experience I had with a client over the weekend.  Let me start the story by telling the ending:

In web development, a launch is not the finish line.  Launching is the starting line.  Not only is it the start of a business, it’s also the start of the next phase of development.  I think the people I advise are a lot more comfortable launching quickly when the realize launching is just the beginning.

The clients I was talking to are trying to launch a relatively straightforward online retail toy store.  The design is beautiful, the product selection exquisite.  But after a year of developing the site (read: one year equals paid lots of money to a web dev shop) the were not very close to launching.

I explained that they needed to focus on launching their site and the way to do that was “strip out everything that is not absolutely 100% essential”.  Like many people launching a web site, they had a million features and loved everyone of them like children…they could not imagine living without any of them.

One of the beautiful things about a web site is that you can regularly and incrementally add features to the site.  It’s not like building a house where it all has to be done at the same time.  For a web site, you can easily build the essentials (kitchen, bathroom, bedroom) then add on the remainder (living room, dining room playroom) later on.

Agile has an expression that I let guide me often: do the simplest thing that could possibly work.  Pause for a moment and think about that: what’s the the absolute minimum necessary?  I wrote to them:

In my opinion, it’s this: home page, product list page, product detail pages, shopping cart, contact info page.  That’s the minimum amount you need to launch an online store.  I would pare it back to that, and launch.

Will that be the most robust and amazing online toy store ever?  No.  Will it work?  Yes.  Does it allow you to start creating value?  Yes.  Does it put you in a position to start learning about your customers?  Yes.  As soon as you launch, can you begin work on some of those amazing toy store features?  Absolutely!

In my view, when you see launching as the beginning, it becomes clear that launching as quickly–even before your product is fully complete or finely polished–is the most logical path to success.

The most dangerous pitfall for a startup is to waste resources–time, energy, and money–on things that don’t matter.  I mean “don’t matter” in a relative, not absolute sense.  In other words, spending resources on things that “don’t matter now“.

When I advise startups I return to this theme a lot.  And, there is one piece of advice that I find myself repeating over and over again: the absolute, number one most important thing is to launch.

Let me repeat that because I can’t emphasize it enough: strip out everything that is not absolutely 100% essential and get your product, feature or site in front of real users or customers.

Why?

In most cases, you are trying to create value with whatever you are launching.  It is impossible for that to occur if you haven’t launched.  Simply put, whatever you are trying to achieve cannot happen until you launch.  Period.  To share the wisdom of a former co-worker, “you gotta be in it to win it”.

Equally as important is that the only way can start learning about what works and what doesn’t is by having real customers use it.  You may think your users or customers want a widget or gidget or flex capacitor, but you just don’t know for certain.  Until you know for certain you’re putting your resources at risk.  The surest way to find out is to get your product in front of the users and learn.

At the end of the day, launching is the name of the game…focus on it like a laser beam and you will succeed.