“We must learn our limits. We are all something, but none of us are everything.”
The internet has reshaped our interaction with users, and our ability to react. It is no longer about the long qualitative loop interaction, but about interacting with people immediately, and in easy to measure ways. It is no longer about needing to treat everyone the same, but about knowing who and where to treat people different. Testing, when done right, allows us to do all of that. It is the process by which we fundamentally grow our understanding of users, of their intentions and of the best way to service them.
It is not about which version of copy grows CTR by 5%.
Business exist to make money and grow. We find new ways to interact with people, find new users, and we find new ways to monetize them. In all of this, we are left to our own talents to make the most of these situations. We use our background, experience, skills, and resources to the best of our abilities, to pick what will move us the farthest and how much we will gain from that action.
We use process to better educate ourselves on the impact of these actions. Whether we do market analysis, KPIs, qualitative, SEM, SEO, ROI, or any other type of processes, these are at best methods to help shape our inputs into the system. At each step, we pick an input (a direction, a new product, a marketing channel) and then calculate the performance only to start the process over again with the next opportunity.
Testing does not magically erase this process, but it does narrow it down. It is no longer about what path to go, it is about which of the paths before you is the best. It is no longer about our experience completely dictating the direction, but about everyone’s voice enabling options which we test to know what the best path is. It is not about starting the process over at each point, but it is now about using the results and numbers to automatically feed the next test, and to continue the growth.
We start with the original point of interaction, and we allow the process to dictate the nest steps. Are we seeing that users respond to images, do we see that they are more likely to spend more time when they use internal search? Do people coming from Google at 4pm on a Thursday respond differently then Canadian users coming at 11am on a Saturday? What on the page matters, what doesn’t? All of these are questions, and from them, we find the answers which spawn more questions. We also find out what needs we are servicing, and for whom, and which ones we are not. The process continues, we start finding multiple options for filling those needs, we start increasing their performance, and we start building this fundamental understanding of what users do, who they are, what they want and what we do to serve them.
It stops being a guessing game. It is no longer about egos or the HiPPo, about marketing versus product versus engineering. It becomes about the user, about their needs and about what we exist to do. All voices start to matter because no voice in the room actually picks the outcome. Performance matters, nothing more, and nothing less.
Being an analyst is not about running tests, looking at numbers, or the tools we use. It is about teaching someone or some group a new decision making process. Data is only data, and questions are only questions, who asks them, who got the data, none of it matters. What matters is our ability to act on them, and our ability to learn from them. Testing, is that process. It is the point where we act, where we enact change and where we create value.
Testing can be done absolutely wrong, and can get you a performance gain. It can be done absolutely correctly, and you may have no options which show an performance gain. The outcome of an individual test, the performance gain, is just the by-product of the process; it is not the end goal nor is it what will grow your company. When I get upset at work, it is never about the outcome, but the process that lead to us getting there. When I am teaching someone about testing, or running a test, it is never about the test itself, but instead about the realization of optimization and the need to add discipline and process to any the decision making process. If I am to be of any value to anyone, it is in working through decision making processes and helping them learn from their own business.
I teach people about acting on numbers. You use metrics to determine current performance, you choose which actions will best grow those numbers, and then you use the same metrics to determine the success of enacting change to those numbers. It is not about the numbers themselves, but about changing them that determines success. Current, short-term fluctuations do not go into this process, nor is success dictated by one day’s data, but by the historical data going forward. In essence, its how were you able to move the needle, and not the actual number. Testing both moves that needle and teaches us how to do it again.
There are major problems that plagues testing, the first of which is that one person or one group thinks that they are the only one who dictates what goes into a test. Testing is the acceptance that you do not know anything, and in most cases know frighteningly little. It is the concept that the option that is best is not what we think, but we can measure and know. It does not come from the largest or the smallest voice in the room, but from any and all voices in the room.
I have stopped trying to count the number of times that the option that everyone thought would win, did win. I have stopped letting the HiPPo eliminate options from a test. I have stopped thinking I can even guess what has a chance of performing. Experience gives me an edge, but the truth is that the style that is thrown in as on a lark seems to win far more often then the one that everyone knows is going to win. The Mendoza line is a great starting point for the accuracy of most peoples ability to gage a test winner going into the initial test.
The other major problem is that most groups are afraid to cause action, thinking that negative results are a failure. In truth, not acting is the ultimate failure, you can always learn from action, but very few people learn from inaction. The key is to have the decision making process in place that best grows whatever it is you hope to accomplish. Action breeds more action. Action may be positive or negative, but if the process is in place, you will see growth over time, and you will learn more about what you do and the people you interact with. That knowledge is golden and is worth more then any tool itself could ever provide.
Testing is the business plan. It is the tool that we enact change with. It is at every step of the process and when done correctly, it is the only thing that gets a say on what actually goes live. Being a part of a test, or of the process, is about taking a step back and removing your ego from the game. It is about accepting that you do not have all the answers, nor does anyone else. It is about allowing peoples skills to shine by opening up their voices and by allowing performance to dictate growth.
