What is the Value of Questions?

 

“If you know the answer, you most likely asked the wrong question”

 

What is life, if not one question that we seek to answer through all of our days?  Questions are the currency by which analyst are paid.  It is not our job to get answers, any person with enough time can find a way to get answers, it is our job, and our quest to ask the right question.  When you walk into a problem, it’s not the how of getting the answer that bothers me, but first figuring out what question to answer.

 

When looking at a property, I am charged with improving things.  That is not normally what is directly said to me, but that is the reality of the situation. Whether it is coming in and looking at SEO, or page testing, or revenue models, or site analysis, the question is never what have we done, but what can we do that will maximize our performance going forward.  There are a million directions that you can choose, but knowing which one, where, how, is what will ultimately drive value for the site.

 

One of the largest dilemmas that I face is when someone will dump a large set of data on me, and expect me to find some needle in a haystack for them.  I always come back to them and work with them to first find the question they seek to answer.  The data has no meaning unless it has context.  The context is the business need, the site, the users, everything.  Understanding it, understanding where we are what we are trying to accomplish, that provides meaning, and only then will the data set provide the slightest bit of value.  Whether it is testing elements on a page, optimizing a single revenue stream, looking for engagement opportunities or looking at general site traffic patterns and KPIs, they all only have value if they are attached to some question.

 

Look at our tools, be it a data solution, data warehousing, tracking, qualitative research or quantitative analysis.  Do any of them actually provide value?  No, they are simply tools by which a user can do something with.  With out the understanding of what it is you are looking for, and the need it meets, these tools by themselves are nothing.  If you have a question, and an understanding of which tools fit which needs, then the answer becomes a matter of practice, and not a measure of skill.

 

What is a KPI but the answer to the question what are the best measures of your growth?  What is a test, but a way to answer which is the best option to get users to do some behavior?  What is the best behavior?  What users?  What is the value of that behavior in the first place?  What is value to this site?  There is nothing I do that does not start with a question.

 

As you can see, the problem with getting answers, is that they drive more questions.  It is a never ending quest, nearly quixotic in nature, to keep finding the best question.  It is this quest that drives me when I wake up, it is this quest that drives me when I go to sleep.  It is the question, and the next question, and the need to know the answer, which defines an analyst, which defines our ability to add value to a property, and to ourselves.

 

 

What is Quantitative Dissonance?

“We can not move forward without a place to start, but we equally can not move forward if we are not capable of trusting our tools to help us.“ 

To operate, we all make a number of assumptions and prejudices to expedite our day in and day our lives.  We use our own history, what we have learned and done, to fill in the gaps of everything around us.  A website operates the same way, with people using their knowledge basis and own instincts to guide and push what they think they see and what they think users are doing.

No one actually knows what people are actually doing and seeing.

We look in aggregate, we look after the fact, our data only shows us what they did do and we can build models to guess what they will do, but ultimately, we are looking into a dark void and trying to figure out what paths lead to safety, and which ones lead us over the precipice. 

Analysis helps mitigate some of this danger, and can help frame where we are and where we thought we were.  This gap, this difference between our preconceived notions of what we think the site does, and what the data tells us the site does, is Quantitative Dissonance.   Both are true, and neither is complete.  Our experiences, our numbers, our people working for us and the people visiting our site, they all shape a vision of what we want to be, and what we are.  It takes discipline, patience, and skill to help close the gap between these points, as well as an understanding that you will never succeed.

Our expectations and experience shape the input of what we were attempting to do in the first place.  Why we put something on a page, or design it some way, funnel users between points A and B, and why we did things in the past.  Even in a perfect world, this is only the starting input for what we will do in the future.  The speed that users and the internet changes makes a single year in this industry move far faster then 10 years and just about all other industries.

So what then helps us tackle this problem?  The simple solution that I shape out to the people I work with goes along these lines:

     1)      Data – Collect as much clean data as possible, but only shape it into paths that give you direct access to the success metrics you shape before hand. 

2)      Bleeders – Evaluate and decide what is failing (not what is succeeding).  Decide if these pare are even necessary.  This step demands brutal honesty and the ability to let go of that which we hold sacred.  This also demands that you know the purpose and strengths/weaknesses of every part of every page you are looking at.

3)      Test! – Start by testing things that you think you need, but see if you do, on what ever pages you users look at most.  Look at long and short term behavior.  Quickly use iteration testing to understand how things actually interact, instead of how we think they interact.

4)      Revaluate – Decide if the pages are doing what you think they should, and decide if everything on them is designed to further that purpose.  If not, go back to step #3 and test to see if a change or removal is possible.  The things we see here that do not match what we thought we knew, this is quantitative dissonance.

5)      Optimize – Once you are comfortable with the numbers, the testing situation, know where you have problems and where you don’t, and can answer whether each piece is driving your site towards its end goal, you can then start to use testing to test out your assumptions and to see how large the gap was/is and could be.  This step never ends and really, should be looked at as the new starting point for everything going forward.

Trusting yourself as much as what the data tells you is the only way to succeed in this pat

As you can see, what we thought we knew acts as the input and the success metrics, but it is the data and the use of it, especially through testing that is what actually shapes whatever element we are working on.  As the steps are followed, you are moving farther and farther away from this ethereal understanding of your site, and you are relying on qualitative and quantitative methods to really direct the next steps, and to find the next question and answer. 

The more voices, the more expertise you have, the better the original input, but the output is a product of discipline and of practice.  If we are not capable of resolving the gap, we will forever be victims of it.