People sometimes try to get too much from experimentation. They think it will do more than it really is meant to do. Experimentation should be considered when we want to compare a small number of alternatives to select the best. Even if we do, experiments don’t normally generate new options and don’t typically optimize anything. They only select the best alternative from among those used.
In marketing, I find other problems in running experiments such as taking the time to conduct them properly. The results are often skewed because of time constraints or just plain laziness. Another problem is getting an adequate sample both in number and similarities to appropriately evaluate the outcome. Most people really do not understand the limitations and if you do, conducting experiments can be much easier and beneficial. In The Market Research Toolbox book the author, Edward F. McQuarrie lists a section of do’s and don’ts for experimentation:
- Don’t be over hasty in arranging to do an experiment. You have to know quite a lot about customers and the market before an experiment can be valuable. If conducted prematurely, you risk getting wonderfully precise answers to the wrong question.
- Don’t let fear of costs prevent you from doing an experiment where appropriate. A laboratory experiment such as the price and product design may cost no more than a focus group study and can be considerably less than traditional survey research with a large national sample.
- Do obsess about getting the details exactly right in your experimental design. The test groups have to be made as equivalent as possible, and that tests stimuli have to differ in precisely those respects, and only those respects, under investigation.
- Don’t be afraid to be a pioneer. Experimentation is one of the several areas of markets research or business to business and technology firms tend to lag behind consumer goods firms as far as best practices concerned.
- Don’t expect brilliant new ideas or standing insights to emerge from experiments. Experimentation does one narrow thing extremely well: it reduces uncertainty about whether a specific message, price change, or product design is superior to another and whether it surmounts a specified hurdle rate. Experimentation is confirmatory market research par excellence; it is not a discovery tool.
Keeping these do and don’ts in mind, should help you get more out of your experiments. Having these boundaries around your expectations can help tremendously in evaluating the outcomes.