What's so great about Benihana, the classic Japanese hibachi restaurant? The food is decent. The chefs know how to juggle knives. And they are able to make normally staid people catch shrimp being flung into their mouths.

These are all neat things, but as the classic Harvard Business School case study points out, the truly great thing about Benihana is that their labor costs are about one-third of the typical U.S. restaurant.

How do they do this? In a nutshell, Rocky Aoki, the founder of Benihana, figured out an ingenious methodology for using people more efficiently. Take the wait staff at Benihana for example. Benihana needs far fewer waiters than the average restaurant because they basically combine multiple tables into large communal tables that are served contemporaneously. Also, since the food is cooked on a stove built right into the table in front of the diners, the waiters don't have to make extra trips back and forth to the kitchen as normally required by a regular restaurant. Thus, the average waiter at Benihana can serve far more people than the average server at a typical restaurant.

What does this have to do with anything? Well the same basic concept is what makes the multivariable methodology for website optimization so special — it allows you to use your audience far more efficiently than other traditional testing methods, giving you the ability to accomplish a lot more with the same number of people. In essence, you are able to spread your audience (your waiters) across a much larger number of simultaneous experiments (your tables).

There is a common misconception that multivariable testing requires a large “sample size,” and that unless you are a high-traffic website, you are resigned to the kids' table that is A/B testing. But that's absolutely not true. The fact is that if you have enough traffic to conduct A/B testing, you typically have enough traffic to conduct multivariable tests. For example, it usually requires the same amount of traffic to test one variable with two values (an A/B test) as it does to test ten variables with two values for each variable. The A/B test will tell you the best solution out of two possibilities. The multivariable test will tell you the best solution out of 1,024 (2^10) possibilities.

The differences can get even more pronounced when you move to the more “advanced” A/B/C test. Depending on how you sequence the A/B/C test, getting significant results on the A/B/C test will actually require at least 50% more traffic than is required in the multivariable test described above.

This is important to understand, particularly if you run a business-to-business site. Don't believe that just because you likely have lower traffic than a typical business-to-consumer site that you can't take advantage of the power of multivariable testing. The key point to remember is this: if you have enough traffic to do an A/B test, you almost always have enough traffic to do a multivariable test.

This month's Customer Success Story highlights such an example. Below we highlight SolidWorks, a highly successful provider of 3D CAD design tools. Being a business-to-business site, they certainly don't have the traffic of a Yahoo!, Google, or Ask.com. But they do have enough traffic to do A/B testing, and thus they have enough traffic to do multivariable testing. According to Alexa, SolidWorks is the 24,920th most popular website. Still, with multivariable testing, they were able to find the best solution out of 18,000 possible versions of a key web page.

So whether you're operating a restaurant or operating a website, it doesn't take any fancy tricks to be successful. You don't need to flip shrimp tails into your hat or make volcanoes out of onions. It just takes a well thought-out methodology that lets you use your resources more efficiently.

To get an introduction to the math of multivariable testing, we invite you to join us at our July General Webinar, “The Simple Math of Multivariable Testing.” To register, please click here.

SolidWorks is a leader in 3D CAD technology, empowering product design teams with intuitive, high performance software that is easy to use, and provides the freedom to design products that set you apart. SolidWorks is 100% focused on product design, and the corporation provides a complete selection of best-in-class software and services to help manufacturers get products to market faster.

SolidWorks brought in Optimost to optimize their e-Drawings home page and increase the number of products downloaded. The dedicated Optimost Client Team developed a test plan for the page, which included 7 variables and 25 different values (versions of variables) to be tested. The Optimost solution engine quickly identified and generated 18,750 template permutations. SolidWorks then conducted multivariable tests on the e-Drawings home page, with a winning creative that increased product downloads by 9.6%.

By making several simple changes to their website, SolidWorks was able to gain significant improvement. Some of the changes included altering the color of the “Download” buttons and the language in the page header.

The optimization experiment also gave SolidWorks great insight as to what worked and what didn't on the page — learnings that will inform future experiments. As a result of this increase in product downloads, SolidWorks is conducting further experiments on their e-Drawings page as well as several other pages on their website.

Q: How can we be sure that dynamic content, such as price variability, doesn't confound the results of our test?

Even if there is a fair bit of price variability (such as in the case of hotel rooms, for example), having adequate sample size will reduce the amount of variation a test page will experience. Below is an example of a creative that runs for one hour (or 10 visitors to the page) and then ten hours (or 100 visitors to the page) during which the price fluctuates. Let's make the assumption that everything else is held constant with respect to the customer (no effects from morning to night, weekend purchasing, seasonality shift, or other external factors). The conversion rate (60%) stays approximately the same between the one hour and ten hour periods of time, but the variation or standard error in the conversion rate is much lower for the longer running period. The standard error calculation can be referenced in the December issue of Optimal InSites. In the case of the one hour period — Table 1(a) — the conversion rate will fluctuate between 45% and 75%. The fluctuation in conversion is solely attributed to the price the customer sees at purchase since we are holding everything else constant during this experiment. If we allow ten hours for the page to gather more sample, additional visitors will reach the page. The more sample accumulated will reduce the large fluctuation (from ± 15% to ± 1.5%) of the conversion rate created by the price variability.

How does this help us?
Now, we can imagine a scenario where we are conducting an A/B test on the above page with such price variability. Assume page B is converting at 70% providing a 16.6% lift over page A converting at 60%. It would be difficult to say that B is outperforming A if the test only ran for one hour due to the large error noted above. Page A's interval of conversion goes from 45% to 75%, encompassing page B's conversion rate of 70% — this would indicate that B may just be converting higher as a result of the influence of price fluctuation. On the other hand, if the test continued up to 10 hours, A's interval of conversion would range from 58.5% to 61.5%. If the improvement sustained itself for page B, the 70% conversion rate would most likely be attributed to the content being served by page B and not any price fluctuations.

In summary, if the margin of improvement is relatively small, a large sample size is required to minimize the pricing effects. To reduce the confounding effects from pricing when sample size is low, the margins of improvement must sufficiently exceed the standard error. The same principles used in the A/B comparison also hold true when conducting multivariable tests where the emphasis on adequate sampling shifts from the creative to the variable level. In addition, the effects of dynamic pricing are analogous to the influence on visitors from rotating promotions on pages, product differentiation on catalog sites and events/news content. Hence, sufficient sampling addresses the same issues that would be created by other sources of dynamic content as well.

Today's industry-leading organizations know that they can only maximize the effectiveness of their websites by properly incorporating the “voice” of their customers. One way that customers voice their opinion is by taking measurable actions on websites which are captured and used by Optimost to identify the most effective web pages.

Bazaarvoice, an Optimost partner, leverages the voice of the customer in a slightly different manner. The company offers outsourced technology, services, analytics, and expertise to help companies enhance the online shopping experience with social commerce applications that drive sales. The company encourages and harnesses word of mouth marketing and brings it closer to their clients’ brand and customer experience.

The Bazaarvoice product suite includes the following solutions:

• Ratings & Reviews (A hosted and managed customer content solution)
• Ask & Answer™ (User generated Questions and Answers)
• SyndicateVoice™ (Uses product reviews as an external tool to drive traffic to your site)

Optimost can also identify the most effective location and presentation of the content generated by these solutions, allowing you to maximize your conversion rates and your online sales revenue. Together, Optimost and Bazaarvoice help you to fully leverage the voice of the customer to optimize your online marketing initiatives.

For more information about Bazaarvoice, please go to www.bazaarvoice.com or send an email to info@bazaarvoice.com.

Click here for a complete listing of Optimost Partners.

If you are an Optimost partner and you are interested in either having your company featured in our Partner Spotlight or your upcoming events listed in our newsletter, please send an email to partners@optimost.com.

Last month, Optimost held its East Coast Workshop in New York, and received a tremendous response from attendees. Megan Burns from Forrester Research delivered the keynote presentation on “Making Testing Optimization Part of Your Marketing DNA,” and Optimost client ThomasNet shared its success story with attendees. Most survey respondents indicated they felt the workshop was valuable and they learned about the benefits of including multivariable optimization in their marketing programs.

In case you missed our East Coast Workshop, there is still time to register for The Optimization Summit, which will be held in San Francisco on October 3, 2007.