Tell me about something that went wrong

Tell me about a project that did not go the way you expected.

The migration we ran last spring. We moved about forty thousand records overnight, and by the morning roughly two per cent of them were wrong.

Two per cent sounds small.

It is, until you consider which two per cent. The failures were concentrated in the oldest accounts, which were also the most valuable ones — so the impact was nothing like two per cent.

What did you do?

We stopped the rollout the same morning, restored the affected accounts from the previous night, and told the client before they noticed. That last part was the argument we had internally.

Someone wanted to wait.

Someone wanted to fix it quietly and report it once it was fixed. I took the view that if we were going to be found out, being found out while telling them was a better position than being found out afterwards.

And with hindsight?

The decision was right. The preparation was not. We had tested against a sample that was broadly representative, and the old accounts were precisely the ones a random sample would under-represent.