Better Answers with Imprecise Data

The ability of market researchers to provide precise forecasts is critical to decision makers. Precise forecasts are very valuable. A 21% share estimate with a confidence interval from 20% to 22% tells you all you need to know if your threshold for launching is 20%. Imprecise forecasts are of less value and sometimes, no value. A 20% share estimate with a confidence interval from 12% to 30% isn’t going to be of much value for making the same launch decision.

Unfortunately researchers often find themselves in situations where it is difficult or impossible to provide precise information. Direct estimates of key variables can be undermined by small samples and/or the method in which data is collected (i.e. qualitative). Other times the estimate needed is the result of several inputs, each with its own range of possible values. When the inputs are combined the range of possible values expands, making the final estimate less precise with each step.

Fortunately there is a way to provide solid forecasting information even when samples are small and estimates have wide confidence intervals. The secret lies in taking a closer look at the range of possible results.

Consider the situation described above – a share estimate of 20% but a confidence interval that ranges from 12% to 30%. Suppose that you knew how often each result between 12% and 30% could occur. If you knew this it would a simple exercise to sum the frequency of every result between your launch threshold of 20% and the upper limit of 30% and calculate what percentage that is of the total. Let’s say it is 60%. This means that based the information available there is a 60% chance of meeting or exceeding the target of a 20% share. Your forecast has gone from an imprecise estimate of share to a precise probability of success that can be used to make the launch decision.

Monte Carlo modeling is a tool that allows researchers to describe and analyze results for forecasts. Monte Carlo models start with an EXCEL spreadsheet that includes all of the variables that can impact the estimate along with the necessary calculations. The variables are represented by probability distributions instead of specific numbers. When a Monte Carlo model runs it randomly samples a number from each probability distribution, calculates the overall estimate and stores the result. It repeats this process thousands of times – essentially taking a large representative sample of the values of each input variable and all the combinations of these variables that could occur. The result is a realistic approximation of the values the estimate could have and how often each could occur.

A recent study we conducted provides a good example of Monte Carlo in action. Our client was assessing the impact of different contracting terms such as WAC price, rebates, risk sharing, etc. with payers to see which combination would be most likely to get them preferred tier placement. They chose to use a conjoint design with a goal of estimating the percentage of patients who would have the drug available on tier two. Unfortunately their budget was limited so the sample for the study was only 25. Needless to say the resulting confidence intervals around the percentage of patients getting the drug on tier two was very wide.

The conjoint simulator for this study was programmed as a Monte Carlo model. The program was much like any other conjoint simulator except that the impact of each contract term had a probability distribution instead of a specific value. When the terms of a contract were entered into the simulator the Monte Carlo program would randomly select a value from the impact distribution of each contract term and use these numbers to estimate the percentage of patients who would get the drug on tier 2. It then repeated this process several thousand times. The result was a realistic distribution of all possible tier 2 outcomes for that contract based on a large representative sample of the range of values for each term and all of their possible combinations.

Our client used the simulator to assess specific contracts that were being considered. They had particular goals for the percentage of patients getting their drug on tier 2 and the Monte Carlo results told them the probability of achieving these goals with each contract. The model also allowed them to compare two potential contracts and identify the one most likely to produce a higher percentage of patients on tier 2.

Over the years we have developed many applications for using Monte Carlo models to meet our clients’ forecasting needs. We have found them particularly useful when the forecast involves integrating information from several sources. For example pricing decisions often require integrating the likelihood of payer restrictions on product use combined with information on how physicians and patients will respond to these restrictions. In these scenarios Monte Carlo not only provides the probability of achieving key share and revenue targets but also can identify the variables that have the biggest impact on outcomes. These variables can then be monitored as lead indicators for in-market success.

Would you like to hear more about how we have used Monte Carlo modeling to provide better information for forecasting? Feel free to email me at bduncan@thinkRGA.com, and we can arrange a time to discuss it further and answer any questions you might have.