Re: How deep do you test?

Just a quick note to follow up my previous post on how deep do you test. I had mentioned some frustration in our test suite regarding integration style tests with asynchronously loaded data. The old adage rings true:

If your tests are hard to write, you're doing it wrong.

I spent the better part of yesterday refactoring the code dealing with the async data after we found that it could have caused some really sneaky bugs. The code is much cleaner now, and well all agreed it was much easier to understand. Everyone was happy and there was much rejoicing.

One last comment before I sign off - the bugs we prevented would have been incredibly hard to track down, and would have likely only shown up intermittently due to cell network latency. TDD really saved us on this one, since without it, we would have never seen the warning signs and realized how fragile some of the logic really was.

Why you should pair program

My new job uses pretty strict XP for software development. I say pretty strict, because as I understand it, they started with strict XP and have adapted it to their needs. To me, coming from a half-hearted Agile/Scrum shop, the two most noticeable features about XP are pair programming and Test Driven Design|Development (TDD).

It seems like, to most people, pair programming is the biggest mystery of the whole process. Before trying it full time, even I was a little bit skeptical as to whether or not it was really worth it. I mean, really, two people doing one job?! ;) (A little background for you non-technical types - pair programming, in a nutshell, is two people sitting at a single computer working on something collaboratively.)

The biggest and most obvious question about pair programming is how can you possibly get as much work done when working in pairs. It's usually phrased something like this:

- Jack can get feature x done in 9 hours.
- Sally can get feature y done in 5 hours.
- Can Jack and Sally get both features done in less than 14 hours by pair programming?

Only having pair programmed for about a week (8 days to be precise), I don't know that I have a good answer for that question quite yet. I also think that the question is going about it the wrong way.

For starters, here's what I have noticed about pair programming so far:

- Having begun a new project on a new framework on a new platform, I came up to speed at least 3-4 times faster than I would have on my own
- Knowledge transfer happens in real time and is unavoidable
- Code reviews are mostly unnecessary
- There are far fewer bugs
- When one of us gets stumped, about 70% of the time the other person has an immediate answer to keep the pair going
- Time wasted trying to run and debug code because of typos, etc. is nearly eliminated
- Time wasted on Digg and Reddit drops to 0
- The job is much more engaging so I don't mind a Digg-less lifestyle
- I make fewer design decisions that I end up having to change later
- My brain hurts when I go home every night, but in a good way

Here are some other reasons why I think the question asking about the 14 hours is overly simplistic:

- The cost of bugs, even ones caught internally, is HUGE
- Code that can be read by 2 people is much more likely to be able to be read by more than 2 people
- Code that can be read by more than 2 people is much easier to maintain
- When a pair says something is done, it's more likely to actually be done, so tracking progress is easier

I think a better way to phrase the 14 hours question is this: Is a team's velocity higher when pair programming? Keep in mind that bugs don't count toward velocity, only new delivered features do.

Even with all these bullet points above, I would guess that pairing is still around 50% faster than doing something on your own. Think about when you're writing code, are you really limited by your typing speed? Most of your actual time is spent thinking about the problem, trying to find the right approach, googling, and testing. Having a pair to work with seriously cuts down on the time required for these parts of development.

So ask yourself, should you be pair programming?

How deep do you test?

I've been talking a lot recently about what the right level of testing is on my current project. Previously, on the Java projects I worked on, I was a full blown mocker, anything but the class I was testing would be likely be mocked and I would try to do pure "unit tests", testing only the code in one class at a time. Mockito was my friend and the world was good.

Now I'm working on a WebOS app so I'm back in the land of JavaScript and dynamic languages. I've never done any real JS testing before (or any dynamic language for that matter), so I'm trying to figure this all out for the first time.

The question of how deep to test keeps coming up because we're running into some some significant test complexity. We're not mocking many of our own classes, only the WebOS classes, so our tests are really halfway between integration tests and true unit tests.

This has its advantages, especially in a dynamic language where parameters are only checked at runtime. If we were doing pure unit tests and mocking the world away, we could be creating a fantasy where our class thinks it's running just fine, but isn't passing the right parameters or could possibly even be calling mocked functions that don't exist.

On the other hand, without mocking, our test complexity goes way up, especially since we're dealing with a lot of async behavior. Consider the following test:

it("should call the callback delegate", function() {
  var foo = new Foo();
  var myCallback = function() {};

  foo.doWork(myCallback);

  expect(myCallback).wasCalled();
});

Here we have a simple case where we are testing a doWork function on the Foo class and expecting that it calls a delegate at some point in the execution.

However, in reality, doWork might call some other class that relies on an XHR and asynchronously loading something from a datastore. Now our test looks something like:

it("should call the callback delegate", function() {
  var foo = new Foo();
  var myCallback = function() {};

  foo.doWork(myCallback);

  Tests.AJAX.Requests.fakeResponseFor(SOME_XHR_REQUEST); // Succeed pending XHR, return stub data
  Tests.Datastore.get.succeedAll(); // Succeed pending async datastore get requests
  Tests.Datastore.add.succeedAll(); // Succeed pending async datastore add requests

  expect(myCallback).wasCalled();
});

As you can see, the test complexity goes up very fast. Almost half the lines of code don't even have to do with the class we're testing!

Mocking could solve this problem much more cleanly by mocking whatever helper Foo uses to do the async logic:

it("should call the callback delegate", function() {
  var foo = new Foo();
  foo.helper = mock('HelperClass');
  var myCallback = function() {};

  foo.doWork(myCallback);

  expect(myCallback).wasCalled();
});

The argument against mocking in the tests above is that it could hide some poor design decisions, especially in our app when dealing with async behavior. For example, you might need a whenReady delegate to execute some code when async data is ready, and if you're mocking everything, that might not be apparent. However, the tradeoff is a lot of test code that detracts from what you are really trying to test.

The option of creating a stub class that automatically succeeds or fails all async behavior was brought up, but the legitimate concern was you could be hiding bigger problems. You could mistakenly try to code it to work synchronously and it would pass in the mocked world, but not in reality.

Also, being true to TDD, we could setup doWork to return a value instead of use a callback. This code would likely succeed with auto-succeed stub classes, but would fail completely when run in the real world:

it("should get a value from #doWork", function() {
  var foo = new Foo();
  var result = foo.doWork(); // doWork will fail in an asynchronous setup

  expect(result).equals('myValue');
});

I could rename this post to "How do you test async behavior?", but I think that my async example is just one of many things that mocks/stubs either help or hurt for the reasons above.

So, how deep do you test your code? Does the language affect your decision? Do you have any general rules of thumb for when to mock or when to do integration tests?

Keeping the knives sharp

For most of my life, my dad ran a successful small business doing hardwood floors. He usually had no more than 4 employees since he thought quality started to suffer beyond that point. Throughout the 25+ years he ran the business, he never did any advertising yet was often booked for months in advance. Word of mouth and repeat customers kept things going strong until he decided to sell the business to a longtime employee and retire.

I was talking to my parents this weekend, and told them I was starting one of these "weblog" things on the "internet". We were discussing the cement cutting board metaphor, and I realized that my dad's business was one of the only places I've ever noticed an absence of cement cutting boards. The way we did things there made sense and were setup to make us do a better job and be more productive.

Talking it over with them, I realized that the biggest factor to preventing the cement cutting boards was that the person making the decisions had hands on time and was a real expert on the tasks we were doing. I couldn't have imagined hearing something like "Go ahead and use that dull sandpaper for a few more days, the new shipment isn't in yet" since the boss knew first hand how unproductive that would have been. The knives (and sandpaper) where kept sharp at all times.

I really can't emphasize this enough: The person making the decisions must be an expert in the domain.

Now, let's apply this to the fun, evolving, and often counterintuitive world that is software development. ManyMost of the things we do don't make sense to people that haven't spent time in the trenches. Think of these things from their perspective:

Unit tests..? You don't even deliver those!

Pair programming..? Two people doing one person's job?!

Behind schedule..? Add more people and work longer hours!

Programmers can't even fully understand those things unless they have experienced them first hand.

Even if the executives or project managers don't have expertise developing software, I would hope the business as a whole has people who do. Use those people! Embrace tools that make sense to the people using them, and don't be afraid to invest in your company. In software, these things often have huge returns on your investment. If a tool that costs $100, even $1000, can prevent one bug from reaching production it will have paid for itself many times over. Keep your developers' knives sharp!

To put it simply, if you can't explain how a good mock framework can cut down on test complexity and increase productivity, should you really be the one to decide whether it should be used?

Hello World! (a.k.a. Enter the Cutting Board)

I know, I know, that's probably the most overused title for a first post. But I'm a programmer... what else am I supposed to say?

I'm starting this blog as I move on to a new job and hopefully plenty of new learning and opportunities. I wanted a place where I could chronicle my thoughts for my own record, and, in the spirit of Atwood, I figured why not put it online in case something interesting comes out of it.

The name of the blog was inspired by recent events in my life, namely many months of using a cement cutting board. Now, this was a beautiful cutting board. Nice and shiny, it was a deep blue gray to match the counters it sat upon. Being cement, the knives used on it didn't leave a scratch and I'm sure it will retain it's shiny surface for years to come.

However, there was a problem with the whole cement cutting board situation - the knives didn't leave a scratch. When your cutting board is harder than your knife blades, pretty soon you have a lot of cooking knives that might as well be giant butter knives. Blades that once cut thin slices off a ripe tomato could now only mangle and eventually smash whatever they were cutting in two.

You might wonder who would buy (or manufacture!) such a thing. After a decent bit of thought I came to only one conclusion - someone who never uses cutting boards. After a bit more thought I realized I was surrounded with cement cutting boards. Software designed by people who never used it. Processes created by people who weren't affected by them. The list could go on and on.

I realized that nearly every institution I had ever known was riddled with cement cutting boards. I began to think it was just how the world had to work due to some unseen saturation of illogical thought. It was as omnipresent as the laws of physics - objects fall when you drop them and you will have to do things that are counter-productive and make no sense.

Yet deep down I knew there must be some oasis of nice, soft, wooden cutting boards, or at least a group of people that despise the cement ones as much as I do. I think I've found such a group, so here begins my journey to see if the grass really is greener on the other side.

- Ian