The internet has reshaped our interaction with users, and our ability to react.  It is no longer about the long qualitative loop interaction, but about interacting with people immediately, and in easy to measure ways.  It is no longer about needing to treat everyone the same, but about knowing who and where to treat people different.  Testing, when done right, allows us to do all of that.  It is the process by which we fundamentally grow our understanding of users, of their intentions and of the best way to service them.

It is not about which version of copy grows CTR by 5%.

Business exist to make money and grow.  We find new ways to interact with people, find new users, and we find new ways to monetize them.  In all of this, we are left to our own talents to make the most of these situations.  We use our background, experience, skills, and resources to the best of our abilities, to pick what will move us the farthest and how much we will gain from that action.

We use process to better educate ourselves on the impact of these actions.  Whether we do market analysis, KPIs, qualitative, SEM, SEO, ROI, or any other type of processes, these are at best methods to help shape our inputs into the system.  At each step, we pick an input (a direction, a new product, a marketing channel) and then calculate the performance only to start the process over again with the next opportunity.

Testing does not magically erase this process, but it does narrow it down.  It is no longer about what path to go, it is about which of the paths before you is the best.  It is no longer about our experience completely dictating the direction, but about everyone’s voice enabling options which we test to know what the best path is.  It is not about starting the process over at each point, but it is now about using the results and numbers to automatically feed the next test, and to continue the growth.

We start with the original point of interaction, and we allow the process to dictate the nest steps.  Are we seeing that users respond to images, do we see that they are more likely to spend more time when they use internal search?  Do people coming from Google at 4pm on a Thursday respond differently then Canadian users coming at 11am on a Saturday?  What on the page matters, what doesn’t?  All of these are questions, and from them, we find the answers which spawn more questions.  We also find out what needs we are servicing, and for whom, and which ones we are not.  The process continues, we start finding multiple options for filling those needs, we start increasing their performance, and we start building this fundamental understanding of what users do, who they are, what they want and what we do to serve them.

It stops being a guessing game.  It is no longer about egos or the HiPPo, about marketing versus product versus engineering.  It becomes about the user, about their needs and about what we exist to do.  All voices start to matter because no voice in the room actually picks the outcome.  Performance matters, nothing more, and nothing less.

Being an analyst is not about running tests, looking at numbers, or the tools we use.  It is about teaching someone or some group a new decision making process.  Data is only data, and questions are only questions, who asks them, who got the data, none of it matters.  What matters is our ability to act on them, and our ability to learn from them.  Testing, is that process.  It is the point where we act, where we enact change and where we create value.

Testing can be done absolutely wrong, and can get you a performance gain.  It can be done absolutely correctly, and you may have no options which show an performance gain.  The outcome of an individual test, the performance gain, is just the by-product of the process; it is not the end goal nor is it what will grow your company.  When I get upset at work, it is never about the outcome, but the process that lead to us getting there.  When I am teaching someone about testing, or running a test, it is never about the test itself, but instead about the realization of optimization and the need to add discipline and process to any the decision making process.  If I am to be of any value to anyone, it is in working through decision making processes and helping them learn from their own business. 

 
I teach people about acting on numbers.  You use metrics to determine current performance, you choose which actions will best grow those numbers, and then you use the same metrics to determine the success of enacting change to those numbers.  It is not about the numbers themselves, but about changing them that determines success.   Current, short-term fluctuations do not go into this process, nor is success dictated by one day’s data, but by the historical data going forward.  In essence, its how were you able to move the needle, and not the actual number.  Testing both moves that needle and teaches us how to do it again.

There are major problems that plagues testing, the first of which is that one person or one group thinks that they are the only one who dictates what goes into a test.  Testing is the acceptance that you do not know anything, and in most cases know frighteningly little.  It is the concept that the option that is best is not what we think, but we can measure and know.  It does not come from the largest or the smallest voice in the room, but from any and all voices in the room.

I have stopped trying to count the number of times that the option that everyone thought would win, did win.  I have stopped letting the HiPPo eliminate options from a test.  I have stopped thinking I can even guess what has a chance of performing.  Experience gives me an edge, but the truth is that the style that is thrown in as on a lark seems to win far more often then the one that everyone knows is going to win.  The Mendoza line is a great starting point for the accuracy of most peoples ability to gage a test winner going into the initial test.

 
The other major problem is that most groups are afraid to cause action, thinking that negative results are a failure.  In truth, not acting is the ultimate failure, you can always learn from action, but very few people learn from inaction.  The key is to have the decision making process in place that best grows whatever it is you hope to accomplish.  Action breeds more action.  Action may be positive or negative, but if the process is in place, you will see growth over time, and you will learn more about what you do and the people you interact with.  That knowledge is golden and is worth more then any tool itself could ever provide.

Testing is the business plan.  It is the tool that we enact change with.  It is at every step of the process and when done correctly, it is the only thing that gets a say on what actually goes live.  Being a part of a test, or of the process, is about taking a step back and removing your ego from the game.  It is about accepting that you do not have all the answers, nor does anyone else.  It is about allowing peoples skills to shine by opening up their voices and by allowing performance to dictate growth.

“If you know the answer, you most likely asked the wrong question”

What is life, if not one question that we seek to answer through all of our days?  Questions are the currency by which analyst are paid.  It is not our job to get answers, any person with enough time can find a way to get answers, it is our job, and our quest to ask the right question.  When you walk into a problem, it’s not the how of getting the answer that bothers me, but first figuring out what question to answer.

When looking at a property, I am charged with improving things.  That is not normally what is directly said to me, but that is the reality of the situation. Whether it is coming in and looking at SEO, or page testing, or revenue models, or site analysis, the question is never what have we done, but what can we do that will maximize our performance going forward.  There are a million directions that you can choose, but knowing which one, where, how, is what will ultimately drive value for the site.

One of the largest dilemmas that I face is when someone will dump a large set of data on me, and expect me to find some needle in a haystack for them.  I always come back to them and work with them to first find the question they seek to answer.  The data has no meaning unless it has context.  The context is the business need, the site, the users, everything.  Understanding it, understanding where we are what we are trying to accomplish, that provides meaning, and only then will the data set provide the slightest bit of value.  Whether it is testing elements on a page, optimizing a single revenue stream, looking for engagement opportunities or looking at general site traffic patterns and KPIs, they all only have value if they are attached to some question.

Look at our tools, be it a data solution, data warehousing, tracking, qualitative research or quantitative analysis.  Do any of them actually provide value?  No, they are simply tools by which a user can do something with.  With out the understanding of what it is you are looking for, and the need it meets, these tools by themselves are nothing.  If you have a question, and an understanding of which tools fit which needs, then the answer becomes a matter of practice, and not a measure of skill.

What is a KPI but the answer to the question what are the best measures of your growth?  What is a test, but a way to answer which is the best option to get users to do some behavior?  What is the best behavior?  What users?  What is the value of that behavior in the first place?  What is value to this site?  There is nothing I do that does not start with a question.

As you can see, the problem with getting answers, is that they drive more questions.  It is a never ending quest, nearly quixotic in nature, to keep finding the best question.  It is this quest that drives me when I wake up, it is this quest that drives me when I go to sleep.  It is the question, and the next question, and the need to know the answer, which defines an analyst, which defines our ability to add value to a property, and to ourselves.

To operate, we all make a number of assumptions and prejudices to expedite our day in and day our lives.  We use our own history, what we have learned and done, to fill in the gaps of everything around us.  A website operates the same way, with people using their knowledge basis and own instincts to guide and push what they think they see and what they think users are doing.

No one actually knows what people are actually doing and seeing.

We look in aggregate, we look after the fact, our data only shows us what they did do and we can build models to guess what they will do, but ultimately, we are looking into a dark void and trying to figure out what paths lead to safety, and which ones lead us over the precipice. 

Analysis helps mitigate some of this danger, and can help frame where we are and where we thought we were.  This gap, this difference between our preconceived notions of what we think the site does, and what the data tells us the site does, is Quantitative Dissonance.   Both are true, and neither is complete.  Our experiences, our numbers, our people working for us and the people visiting our site, they all shape a vision of what we want to be, and what we are.  It takes discipline, patience, and skill to help close the gap between these points, as well as an understanding that you will never succeed.

Our expectations and experience shape the input of what we were attempting to do in the first place.  Why we put something on a page, or design it some way, funnel users between points A and B, and why we did things in the past.  Even in a perfect world, this is only the starting input for what we will do in the future.  The speed that users and the internet changes makes a single year in this industry move far faster then 10 years and just about all other industries.

So what then helps us tackle this problem?  The simple solution that I shape out to the people I work with goes along these lines:

     1)      Data – Collect as much clean data as possible, but only shape it into paths that give you direct access to the success metrics you shape before hand. 

2)      Bleeders – Evaluate and decide what is failing (not what is succeeding).  Decide if these pare are even necessary.  This step demands brutal honesty and the ability to let go of that which we hold sacred.  This also demands that you know the purpose and strengths/weaknesses of every part of every page you are looking at.

3)      Test! – Start by testing things that you think you need, but see if you do, on what ever pages you users look at most.  Look at long and short term behavior.  Quickly use iteration testing to understand how things actually interact, instead of how we think they interact.

4)      Revaluate – Decide if the pages are doing what you think they should, and decide if everything on them is designed to further that purpose.  If not, go back to step #3 and test to see if a change or removal is possible.  The things we see here that do not match what we thought we knew, this is quantitative dissonance.

5)      Optimize – Once you are comfortable with the numbers, the testing situation, know where you have problems and where you don’t, and can answer whether each piece is driving your site towards its end goal, you can then start to use testing to test out your assumptions and to see how large the gap was/is and could be.  This step never ends and really, should be looked at as the new starting point for everything going forward.

Trusting yourself as much as what the data tells you is the only way to succeed in this pat

As you can see, what we thought we knew acts as the input and the success metrics, but it is the data and the use of it, especially through testing that is what actually shapes whatever element we are working on.  As the steps are followed, you are moving farther and farther away from this ethereal understanding of your site, and you are relying on qualitative and quantitative methods to really direct the next steps, and to find the next question and answer. 

The more voices, the more expertise you have, the better the original input, but the output is a product of discipline and of practice.  If we are not capable of resolving the gap, we will forever be victims of it.

The world of an analyst is tied to the numbers they interact with, the tools that capture them, and the actions we take against them.  Even with all of that energy and the layers on which that defines the role and the world that they live in, numbers are completely meaningless.

I am not paid to find numbers, I am not paid by which numbers are showing.  I can say any number in the world to any other person, and they will just look at me like I have lost my mind.  It is also not the tools we gather them with or how we track them.  I do not care if you use Omniture, Web Trends, Google Analytics, Cognos, or any of the thousands of other offerings, they are just tools.  There are differences between them, there are strengths, weaknesses, uses and reports, but they are just shiny hammers who are only as good as the people who swing them and only can be used for certain functions.

Insight has value.  Being able to find context, both from the tangible and the ethereal, is what matters.  It’s not the numbers, but the knowledge of what they really mean and being able to act on them that matter.  It is not what we hold true, or what we have done, or where we are.  It is what we are doing to move towards the end goal and to make sure that we are where we want to be.  Our insights, our values, shape what we see and what we do, but if you are not able to step back and get a read on where you are going, be it through quantitative or qualitative methods, then you are not anywhere.  A map is only useful with a start and a finish, if there is no context, it is simply lines on paper.

This falls back to the classic  KPI discussion.  Many companies are great at creating dashboards, or finding metrics, or counting things.  None of these have any real use, other then to make some executive feel better about themselves and give them more things to hide behind.  Value comes from knowing what adds value to your path, and what doesn’t, and knowing what actions you can take to drive that value.  If you can not find the few meaningful value adding things you must do, and measure yourselves against them, you will never succeed.  This is not the number, but the actions.  The insight will reflect the action, and will measure it, but it is never a replacement for that action, nor is data the insight that you may gain from it.

My world mostly focuses on media and content websites, though I have touched just about every single type of business model online that there is.  It is fundamental to know who is looking at what, where, and how.  But that doesn’t help shape where to go, how to get these people to stay, or what value they really add to the bottom line. 

The truth is, that almost no one knows what they are really doing.  They think they do, they think they can guess which way the wind is blowing, how they got to where they are, and what they need to do to impact the fundamental equation of their business.  The truth is they are nearly blind and falling into the darkness every moment.  That is why it is so important that you stay focused on the few, simple actions, that you must do to drive forward.  These shining lights will always be there, and they will help you see the world around you.  You won’t see everything, and sometimes we are scared by how little we do see, but in the end, the only light we have we have to make for ourselves.

When I step into a room to work with different groups, no matter the function or hat I have on, the most important thing I must do is understand their business needs.  I often times have to understand them far better and more meaningfully then the people who have done this for years.  This isn’t because I know more then they do, but because I practice the discipline every day to look at a site or function and to narrow it down to key actions which will drive it.  Without that understanding, you will never be able to drive successful change, leaving both sides with wasted energies and lost time.

Data is my divining rod, it is how I look at the world.  I can hold a metric, or a number in my hand, but without the ability to tie it to the tangible, and to understand what it is that got it to where it is and what actions are needed to change it, it has no value.  Others have their own barometers, and they are all correct.  It’s only the ability to take that element, to understand it and act on it, that really matters.

“History shows that those who are best able to utilize their resources towards their goal are the winners.  Those who do not are left as afterthoughts in pages for people who study but do not learn.”

What is a website? Not in the physical or mental sense, but what is it as a function?  What is the purpose of putting something out for other people to find and interact with?  If you can’t answer this, if you aren’t absolutely clear about what you are doing it, why, and how you are going to reach that end point, then you are failing before you even start.

Like Joseph Campbell views stories, I view the internet.  There are thousands of ways to go about it, but in the end, its always the same story.

You have users (X), they do something or many things, to achieve the outcome (Y).  Y can be many things, in the commercial sense, it is the dollar figure we put on the outcome of a site interaction.  For personal blogs, it is in the mindshare that we are attempting to achieve.  It’s not a discussion of success, it is a discussion of outcome.  For every X, there is a Y.  To achieve a set Y, we have two options, increase X, or find the levers that positively change the formula to get more Y from X.  Success is not doing one of those two options, but doing both, and knowing which one you are doing at any given time and being able to balance them.  That is managing a website.

We like to think we know what they do, and sometimes we have good insight into it, but we really don’t have a complete picture, nor is it even possible to have one.  What we are left with are insights and pictures into a constantly shifting playing field.  What is true today will not be true tomorrow.  What we think we know, we never do.

There are many ways to try to get insight, but my world resolves around numbers.  How do we measure what is there, how does that match against what we think we know is “true”, and what is the best way in which to enact change?  It’s a daily battle, but it’s the only one worth fighting. 

To be successful, all you have to do is to be able to find the best ways to leverage your available tools and skills and enact positive change towards some end goal.  You need to have something you, your team, your site, your company are working towards, and you need to focus everything towards that goal. Not on a yearly basis, not on a weekly, or daily, but minute by minute, hour by hour.  The key is not to do it once, but to make every moment about that.  Failing is when you lose focus or are not willing to make the sacrifices necessary to reach that end goal.  Every step of the way, the energy needed to stay on point becomes harder and more difficult, which means that each level requires more sacrifice and more discipline.

In all of this, there is not discussion about your personal skill set, or department, or experience.  Those help you fill in the gaps and the roles that are needed to reach that end point.  The truth is that the only thing that matters is putting every bit of energy towards that end goal, and to refocus and put more energy, more thought to achieving it.

If you can do that, if you can find a way to gain insight (numbers are not insight!), then skill and talent will be able to go their natural course.  If you do not have that insight, that vision, and the ability to focus every moment of every day, all the talent in the world will not let you “win” in the end.