Reconsidering the Value of Small Samples in Marketing Research

Do you remember the famous line from Animal Farm, “Two legs bad, four legs good”?  I suspect that we quantitative researchers have a similar one that goes “small samples bad, big samples good.” Small samples make us nervous. We can’t believe they are representative or that they can capture the diversity of a large population.  We have visions of extreme groups dominating the results. When we hear “small sample,” we fear the outcome will be misleading. The time has come that we should be reconsidering the value of small samples in marketing research.

At least some of our concern stems from a misunderstanding of the term, “representativeness.”  When sampling experts talk of representativeness they are thinking about sample selection, not sample size. To represent a population, a sample must give all members of the population an equal chance of participating. Random samples and related offshoots (i.e. stratified samples etc.) do this and so represent the gold standard.

Unfortunately, practical limitations of time, budget, and sample availability usually make true random samples impossible for market researchers. So instead we approximate a random sample by inviting a diverse subgroup of our target population to participate and argue that the subgroup and hence the resulting sample is a good representation of the population of interest. That may or may not be so. However, the key point is this – if a small sample is selected from a population in the same manner as a large sample, one is no more representative than the other.

The ability to capture the diversity of a population is another matter. Larger samples will capture more of the diversity of a population than small samples. If there are fifty ways to leave your lover, as Paul Simon sang in his famous song, a sample of 20 is not going to identify them all but a sample of 100 might. The bigger issue is – do you care?

Market research typically focuses on the majority perspective and that of meaningful segments. 5% of the population with an unusual perspective might be of intellectual interest but is usually too small to have practical value. Marketers are much more likely to focus on segments of 20% or larger.

As segments get larger, the advantage of large samples in capturing diversity quickly dissipates. Recently I posted an article for the RG+A blog describing how qualitative research can capture segment perspectives. The basis of the article is a series of calculations showing the probability of having at least 3 respondents from different sized segments in samples from 10 to 30 people. The surprising result is that samples as small as 30 have over a 90% probability of having at least 3 respondents from segments as small 20%. In qualitative research with the ability to probe and adapt questions this is enough to identify potentially meaningful viewpoints.

Extreme group dominance is also less of an issue for small samples than you might think. All samples carry a risk of a subgroup dominating the sample and severely distorting any population estimates. The risk is larger with small samples but still surprisingly small. As a simple example, innovators and early adopters make up approximately 16% of a population. Tripling their size would severely distort results, but with a sample of 30 that would only happen twice in every thousand studies.

As small as the risk of extreme group dominance is, it also can be easily negated by including confidence intervals around any statistic computed from the sample. Confidence intervals account for the risk of extreme group dominance as well as any other distortion that might occur due to a small sample size. Confidence intervals also provide needed perspective for decision makers by reminding them that their statistic of interest could have a range of values.

So what does all of this mean? Quite simply, you can learn a lot from a small sample. Their results are not as precise as larger samples, but they can still provide a reasonable description of a target population. They also have the advantage of being faster and less expensive. Dollar for dollar, small samples often provide a better return on your market research investment than larger samples.

We have conducted many small sample studies at RG+A. Several have been with payers, a group comprised of a relatively small number of decision makers who are often difficult to reach. We have also found applications with physicians assessing licensing opportunities, orphan drugs and early stage products. Finally, we have completed several qualitative conjoint studies based on small, nationally recruited samples. The resulting combination of “hard data” and qualitative insights suggests a range of applications that will further enhance the value researchers can get from smaller samples.

Would you like to hear more about where we have found small samples to be particularly valuable?  Feel free to contact me at bduncan@thinkrga.com.