Confirmation Bias and Tracks In The Snow

posted by: Nick Roman on 4/27/2010

An interesting post in the Harvard Business Review blog caught my eye this week.  In it the author discussed the concept of Confirmation Bias, in which we tend to see those things we want to see in data.  In his case, he managed to nearly give himself frostbite, simply because he refused to credit the evidence in front of him that he was underdressed while skiing.  The fact that it was spring and should have been warm kept him from protecting himself against the very cold day.

One of my favorite baseball writers, Bill James, has another way of expressing the same idea.  He has made a career of debunking carelessly advanced theories of baseball, writing that if those theories were correct there would be ample statistical evidence to back them up.  In James’ parlance, those elephants would leave “tracks in the snow” that one could follow.

Do you find that you use customer insight data to reinforce preconceived ideas about your customers rather than take a rigorous view of the data in order to challenge those preconceptions?

An effective technique to provide this exacting view is to ask a set of questions that allow you to identify the drivers of KPIs such as revenue, customer satisfaction, or advocacy.  Correlating the responses to those business metric drivers helps you to understand on which of those drivers you should focus efforts in order to increase performance of those metrics.

A customer of ours, Egg, a wholly owned subsidiary of Citigroup, decided to understand the underlying factors that move customer satisfaction with their contact center.  Much has been written about the quality of contact center service in the UK, and although Egg had a very good reputation (maybe because they had a good reputation!), they decided to look at their preconceptions about the investments and process changes they would need to make in order to increase contact center satisfaction and agent performance.

Contact center management was convinced that the most important factor was the amount of time customers spent on hold waiting to speak to a live agent on the phone.  When they added to their customer surveys a question asking customers to agree or disagree with a series of statements, they found that agreement with the statement “I quickly got through to a live person” correlated least closely with customer satisfaction.

In fact, agreement with the statement “I felt my time was well spent” correlated most closely to satisfaction.  In other words, “I care less about how long it takes to get to a live person than whether that person can solve my problem.”

These results encouraged Egg to move from a costly decision to invest in a new contact center to a decision to invest in “Customer Voice Analysts” who coached agents to be more effective in solving customer problems.

Warning: Correlation doesn’t imply causation.   For instance, it’s often posited that happy customers create happy employees.  Or is it that happy employees create happy customers?  Whichever, it tends to be clear that the two go together.

In Egg’s case, the tracks in the snow led to the right investment decision.  Their Voice of the Customer program led to their asking customers whether their agents resolved their problem on the first call (First Contact Resolution score), which led both to increased customer satisfaction and increased agent satisfaction.

Where do your tracks lead?  Do they confirm what you already think you know, or do they take you in a completely different direction?