What is Quantitative Dissonance?
Posted by antfoodz on March 31, 2008
“We can not move forward without a place to start, but we equally can not move forward if we are not capable of trusting our tools to help us.“
To operate, we all make a number of assumptions and prejudices to expedite our day in and day our lives. We use our own history, what we have learned and done, to fill in the gaps of everything around us. A website operates the same way, with people using their knowledge basis and own instincts to guide and push what they think they see and what they think users are doing.
No one actually knows what people are actually doing and seeing.
We look in aggregate, we look after the fact, our data only shows us what they did do and we can build models to guess what they will do, but ultimately, we are looking into a dark void and trying to figure out what paths lead to safety, and which ones lead us over the precipice.
Analysis helps mitigate some of this danger, and can help frame where we are and where we thought we were. This gap, this difference between our preconceived notions of what we think the site does, and what the data tells us the site does, is Quantitative Dissonance. Both are true, and neither is complete. Our experiences, our numbers, our people working for us and the people visiting our site, they all shape a vision of what we want to be, and what we are. It takes discipline, patience, and skill to help close the gap between these points, as well as an understanding that you will never succeed.
Our expectations and experience shape the input of what we were attempting to do in the first place. Why we put something on a page, or design it some way, funnel users between points A and B, and why we did things in the past. Even in a perfect world, this is only the starting input for what we will do in the future. The speed that users and the internet changes makes a single year in this industry move far faster then 10 years and just about all other industries.
So what then helps us tackle this problem? The simple solution that I shape out to the people I work with goes along these lines:
1) Data – Collect as much clean data as possible, but only shape it into paths that give you direct access to the success metrics you shape before hand.
2) Bleeders – Evaluate and decide what is failing (not what is succeeding). Decide if these pare are even necessary. This step demands brutal honesty and the ability to let go of that which we hold sacred. This also demands that you know the purpose and strengths/weaknesses of every part of every page you are looking at.
3) Test! – Start by testing things that you think you need, but see if you do, on what ever pages you users look at most. Look at long and short term behavior. Quickly use iteration testing to understand how things actually interact, instead of how we think they interact.
4) Revaluate – Decide if the pages are doing what you think they should, and decide if everything on them is designed to further that purpose. If not, go back to step #3 and test to see if a change or removal is possible. The things we see here that do not match what we thought we knew, this is quantitative dissonance.
5) Optimize – Once you are comfortable with the numbers, the testing situation, know where you have problems and where you don’t, and can answer whether each piece is driving your site towards its end goal, you can then start to use testing to test out your assumptions and to see how large the gap was/is and could be. This step never ends and really, should be looked at as the new starting point for everything going forward.
Trusting yourself as much as what the data tells you is the only way to succeed in this pat
As you can see, what we thought we knew acts as the input and the success metrics, but it is the data and the use of it, especially through testing that is what actually shapes whatever element we are working on. As the steps are followed, you are moving farther and farther away from this ethereal understanding of your site, and you are relying on qualitative and quantitative methods to really direct the next steps, and to find the next question and answer.
The more voices, the more expertise you have, the better the original input, but the output is a product of discipline and of practice. If we are not capable of resolving the gap, we will forever be victims of it.
