On my most recent trip to Japan, I marveled at how the internet had made being away from home so much easier. As I read box scores on ESPN.com, while listening to streaming audio on Yahoo! Music, and IMing with my 6-year old daughter, it struck me that the disorienting Lost in Translation feeling of being disconnected from home was probably gone forever.

It wasn’t always that way. I remember my first business trip to Japan when Amazon was just a river and Google was just a spelling error. While it was fascinating to be in Tokyo, it was also very lonely with nothing remotely recognizable on television, and no friends or family within 6,000 miles. I needed a reminder of home, so I did what any self-respecting pop culture addict or Super Bowl-winning quarterback would do. I decided, "I’m going to (Tokyo) Disneyland!"

For anyone who grew up going on annual winter vacation trips to Disney World in Florida, Tokyo Disneyland is like an out-of-body experience. There is not a Westerner in sight, no one is speaking English, yet you feel like you’ve been there a thousand times before. The layout is identical. The rides are identical. The lines are identical. And just like its counterpart in the US (and Paris), it’s the most popular amusement park in the country.

Sure, there are differences. In Disneyland Paris, they serve alcohol. In Disneyland Tokyo, they serve sushi and curry-rice. But the underlying methodology is the same: themed lands, autograph-signing characters, great customer service, and a giant castle in the middle of it all.

Like the Disney methodology, a multivariable testing methodology online is proving that it can work anywhere in the world. As we’ve discussed in previous newsletters, a common finding in the multivariable tests we conduct is that very subtle differences in variables, like headlines, layouts, and submit buttons, can make very significant differences in response rates. Until about a year ago, most of our work was done for US-based companies. But in the past year, we’ve seen the exact same phenomenon in other countries and languages.

Here’s an example from Transcosmos, our partner in Japan:

The image on the left shows the original control homepage for Credit Saison, a financial services company in Japan. The image on the right shows the winning version after multivariable testing. From a distance, the two pages look similar, but a closer inspection reveals many seemingly minor differences in design, layout, and copy. The result was anything but minor; Credit Saison was able to achieve a 22.6% improvement in conversion between the winning version and the control.

With the opening this past month of the Optimost Europe office in London, we’re seeing more and more every day how multivariable testing knows no boundaries. We have already seen impressive results in Spain, Germany and other parts of Europe.

As with Disney, the specifics of what actually works best in different countries will vary. But how to get at those answers, the best methodology for getting at those answers, is consistent. Whether you’re optimizing in the US, or optimising in the UK, you’re going to find that multivariable testing speaks your language.

Delta Air Lines is one of the largest airlines in the world offering flights to 453 worldwide destinations in 97 countries. Millions of customers each day rely on delta.com to research flights, purchase tickets, and manage itineraries.

Delta’s online purchase process consists of five steps:

Step 1: Enter Flight Information Step 2: Select Flight Step 3: Enter Personal Information Step 4: Verify Information/Purchase Step 5: Confirmation An analysis of the site traffic revealed that a large percentage of web visitors dropped off after reaching Step 3, the page that requires them to enter their personal information. This drop-off was especially confusing to Delta since visitors reaching this stage have already seen pricing and made their flight selections. Realizing that changes needed to be made, Delta brought in Optimost to help analyze their sales funnel, conduct online tests as needed, and ultimately increase their online revenue.

The dedicated Optimost team conducted multivariable tests on the Step 3 page. Based on the test plan, which included 8 variables and 52 values (different versions of the variables), the Optimost engine was able to quickly identify and generate over 1.4 million possible permutations of the page.

In addition to testing a wide range of variables like the passenger information fields, instructions, and billing information, Optimost was able to conduct segmentation tests, looking at conversions during weekdays, weekends, and different dayparts. Based on the success of this initial test, Delta has dramatically increased the scope of its testing program and is now conducting a series of multivariable tests on multiple pages throughout their site.

Our next customer-only Webinar will further detail the proper approach for testing an online sales funnel and will explore real-world examples of how other industry leaders have used multivariable testing to optimize their online processes. Click here to register and get find out more information about this event.

You can access all of our customer success stories at: http://www.optimost.com/case_studies.phtml.

Q: What is regression toward the mean?
The phenomenon of regression toward the mean (average) occurs in all areas of testing. Essentially, it describes the natural tendency of a measurement to move closer to its actual true mean. For example, measuring a patient’s blood pressure may yield a high number in the beginning of an exam (this could be due to chance, person’s state of activity or stress, etc.). Over time, re-testing the patient’s blood pressure will yield measurements that may actually be lower than what was previously taken. Hence, the patient’s blood pressure regressed back toward its true reading. The same could hold true for a measurement that may have been unusually low. In that case, with more retesting, we could verify that the patient’s blood pressure is actually higher than previously sampled.

This analogy holds true for testing variations on web pages. Instead of taking blood pressure readings, we are tracking conversions for different web pages. Figure 1 shows a bell shaped distribution that represents randomly designed web pages where the average conversion rate is 10%. After a week of running the test, web page A is observed and its conversion rate is 14%. As the test continues, web page A is observed over time. Eventually, it settles back closer to the overall average. It’s possible that A’s content was getting an unusually high number of conversions solely due to external and uncontrollable factors. Though at first it may appear that A is significantly better than the overall average, overtime with more sample, A’s actual conversion rate of 12% is lower than what was previously sampled and resides amongst the bulk of randomly produced web pages.

For the same test, an optimized web page B is introduced. B has been optimized to contain the best content and is tested with the rest of the randomly designed web pages. Figure 2 shows its performance. Currently, web page B is converting at 16% which is a 6% margin of improvement. Since the overall average is 10%, we would expect over time and with more sample for B to move closer toward that overall average simply due to regression toward the mean. With additional sample, we would eventually see B move back and settle at its actual conversion rate, which in this case is 14%. The true margin of improvement due to the optimization is 4%.

It’s important for future testing to distinguish between tests whose lift does not sustain as a result of unreliable content tested versus lifts that have decreased simply to do a regression toward the mean.

1. The margin of improvement may decrease because the supported content tested never yielded reliable or statistically significant results on conversion rate. Regression toward the mean will be present, but the most important cause of decrease in this case is weakly supported content.
2. The margin of improvement is a result of content tested and shown to produce statistically significant and robust impact on conversion rate. Expect lift to decrease solely due to regression toward mean.

As we have mentioned before in past newsletters, smart marketers are looking to solve the "X times Y" equation, where X is getting the maximum number of visitors to your site for the least amount of marketing dollars, and Y is maximizing the percentage of those visitors who become customers. Optimost clients are maximizing the conversion rates on their websites (the "Y" part of that equation), and Optimost partners with a number of companies that focus on "X", getting those visitors to your site in the first place.

One of those partners is Reprise Media, which offers clients a complete managed solution for their search marketing needs focused on improving their advertising ROI. The company delivers services and technology that enable advertisers to manage and optimize their customer acquisition programs through all search, auction-based and category targeted digital media. Forrester Research referred to Reprise as “the biggest breakout star” in a recent report about the SEM industry.

Reprise Media offers services in the following areas:

• Paid Search Marketing – Reprise helps its clients easily manage search campaigns, improve search campaign ROI, and get better visibility into campaign performance
• Organic Search Optimization – The company makes it easier for spiders to detect and "crawl" pages, increases page rankings, and brings more of the right traffic to client websites
• Analytics and Insight – Reprise provides its clients with timely insights and analysis required to make informed decisions and take appropriate actions to capitalize on business opportunities

For more information about Reprise Media, please go to www.reprisemedia.com or call 212.444.7599.

Click here for a complete listing of Optimost Partners.

If you are an Optimost partner and you are interested in either having your company featured in our Partner Spotlight or your upcoming events listed in our newsletter, please send an email to .

Optimost has scheduled its inaugural East Coast Workshop for Wednesday, May 2, 2007 at the Grand Hyatt Hotel in New York City. This free event will address key issues and challenges in your online marketing environment, and detail how online testing can help you maximize the overall ROI of your online marketing initiatives. Stay tuned for more details!

For more information on the Optimost East Coast Workshop, please contact us at

Seth Rosenblatt, the VP of Marketing and Business Development at Optimost, will be interviewed on the "Online Marketing with RSS Ray" Radio Show. The interview topic will be "Measurable Results with Online Marketing" and will take place on Wednesday, April 4, 2007 at 1pm ET/10am PT.

For more information about the show, please visit the RSS Ray website at www.rssray.com.

A growing percentage of the millions of viewers who watch the Super Bowl are focused on one thing: the commercials. The Monday after the game, you’re likely to find more coverage and critiques of the ads than you are of the actual game in many media vehicles.

Optimost and Reprise Media have joined forces to develop a unique scorecard for advertisers to assess how well they leveraged and integrated search marketing programs and landing pages with their Super Bowl TV commercials. In addition to the scorecard, a whitepaper that includes a detailed online marketing assessment of all the commercials is now available.

For more information, please check out the Super Bowl Scorecard, and go to the Scorecard information page to register online to receive the white paper.