• Change only one variable at a time. This is a cardinal rule for testing any kind of change. That's because if you change more than one variable at a time, you are unable to determine which variable is responsible for the change and to what degree.
• Learn the precise process for diverting traffic. One of the problems in A/B testing is that marketers don't fully understand the mechanism for diverting traffic, thus are not getting accurate traffic numbers. There are several strategies that can be used to divert traffic, but these are not well understood by the layman. The object of the traffic diversion mechanism is to redirect a known percentage of visitors through a modified process. In the ideal situation, the percentage of traffic to be redirected can be easily changed without having to radically modify pages.
• Establish accurate measures of volume: Start by getting a "visitors per page" count from your web analytics tool as step 1 in A/B testing. This ensures that you actually get the percentage of traffic moving through your funnel that is expected based on the number of changes tested. For instance, if you are running 50/50 through A and B, you should see near equal numbers of visitors to the first page in the process and continue running the test until the numbers have converged to about 50 percent through each path. If it's a three-way test, you should see a distribution of 33/33/34 percent of visitors running through each path.
• Look for significant differences: If you see a difference in the conversion rate for the B test, you need to ensure this difference is significant (>.05) so it can be attributed to the step changed in the sales process. Smaller differences can be due to variations in visitor quality (noise). Run your test until all of the change can be attributed to the exact step that was modified, or until you are certain there was no change.
• Take the time to do a null test: Before A/B testing, run a null test by flowing 50/50 traffic through the exact same pages to be A/B tested. This will verify that you get the same conversion and abandonment rates, and that your measurement tools are set up correctly. If you are not getting close to the same rates (within .05) for both tests then something is wrong, and your data from the A/B test will not be valid. If this problem arises, check that (1) you are sending visitors into the tests exactly the same way, i.e., visitors are not pre-selected, sending more qualified prospects down one path or the other; and (2) you are running enough visitors through the test. Depending on your traffic volume, you need to attain a reasonable sample, and this can take time. But it is worth the effort to run a null test because you will have more confidence in your A/B test data.
• Run your test long enough to ensure results are real. The most common mistake in A/B testing is not running the test long enough to determine actual differences. There may be trends in the first few hours that will reverse themselves later. You need a representative sample before you can assume that B is better than A or vice versa. For information on what constitutes a statistically relevant sample consult a statistics textbook.
• Run segmentation tests: Your analytics solution should provide robust segmentation tools. Segmenting your test subjects will allow you to monitor their activities when they return to your site. This lets you pinpoint a possible high-value group of visitors if it turns out that a good percentage of your B-test visitors (A/B test results favored B over A) returned to the website within two months to make another purchase. You can also compare this information to your site-wide loyalty metrics for a better understanding of the long-term effects of your A/B test.

In the past, few marketers were tracking conversions, much less return on investment (ROI) for marketing initiatives. In fact, many marketers thought search marketing campaigns were too hard to measure. A 2003 industry study reported that 3 in 10 marketers were not measuring search campaigns, while 4 in 10 were using outdated web metrics such as click-throughs.

A 2005 search metrics study reveals that nearly 1 out of 2 search marketers who outsource SEO and paid search services either do not elect to measure ROI, are unable to calculate ROI, or can't distinguish between SEO and paid search ROI. This result was obtained despite the fact that most vendors provide access to web analytics tools. One would think that marketers would be interested in validating their vendor's performance by quantifying results.

Demand for Marketing Accountability

You've all heard the famous expression, "I know that half of my advertising is wasted - I just don't know which half." It's been attributed to retailer John Wanamaker and Unilever founder Lord Leverhulme. Marketing has come a long way since then, especially since the advent of online marketing and the current proliferation of web analytics tools.

While marketers are poised to take a quantum leap in marketing accountability with the availability of the new web analytics tools, few of them have these tools in place to measure their marketing performance. Top level company executives are demanding more accountability from their marketing departments and few CMOs can deliver.

An ANA survey of national marketers conducted this year revealed that the majority of respondents (61.5%) believe advertising accountability should be defined, measured and implemented. However, less than one-fifth (19%) are satisfied with their ability to do so. It sounds like very little progress has been made since 2002, when eMarketer reported that 56 percent of marketers did not have the tools in place for measuring marketing performance.

Even today, marketers do not have complete confidence in measuring web marketing results. Industry research reports that 43 percent of the marketing professionals surveyed indicated they were "fairly confident," while only 5 percent were "very confident". The rest were split, with 26 percent being "moderately confident" and 26 percent admitting to "flying blind."

Once you finished the process, the only thing left to do is to start it all over again! Continuously monitoring your conversions and adjusting your goals and metrics will allow you to stay ahead of your competition. They say knowing is half the battle and this methodology is designed to get you there. By studying your data and implementing different strategies, you will be able to identify the areas that need improvement to better maximize your ROI. Like any other area related to internet marketing, the best way to be successful is with continual refinement. Once you've reached your goals, go back and make new ones. Your reached goals are your new baselines, there is always more to achieve, so continue to set the bar higher.