1 / 95

Chapter 13

Chapter 13. Assessing Marketing Test Results. Objectives. Assessing marketing test results based on confidence intervals and hypothesis tests The Central Limit Theorem How to set the confidence level of your tests Making a business decision based on the results of your tests

craig-sparks
Download Presentation

Chapter 13

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Perry D. Drake, Drake Direct

  2. Chapter 13 Assessing Marketing Test Results Perry D. Drake, Drake Direct

  3. Objectives • Assessing marketing test results based on confidence intervals and hypothesis tests • The Central Limit Theorem • How to set the confidence level of your tests • Making a business decision based on the results of your tests • Gross versus net • Multiple comparisons • Calculating breakeven response rates • Facts regarding confidence intervals and hypothesis tests

  4. Introduction - Why, What, How & When We Test Why we Test • Testing is the foundation upon which one builds and grows a direct marketing firm. • With a database, names can be selected for certain treatments and comparisons on the customer’s reaction to these treatments made. • Based on these results, in conjunction with marketing cost and revenue figures, the most profitable decision can be made.

  5. Introduction -Why, What, How & When We Test (Continued) What we Test The types of things most often tested by direct marketers include: • Offers • Creative • Promotional Formats • Outer Envelopes • Copy • Lists • Customer Segments • Payment Terms • Guarantees • Premiums

  6. Introduction -Why, What, How & When We Test (Continued) How we Test • We test a new format, creative concept or offering by selecting a sample of names of interest. • This sample is a subset of customer records and a random selection from the universe of interest on the database.

  7. Introduction -Why, What, How & When We Test (Continued) How we Test (cont.) As mentioned back in Chapter 6, to ensure the test results are meaningful, the sample must be representative of the entire population of concern. • A representative sample is a sample truly reflecting the population of interest from which the direct marketer draws inferences. • For a sample to be representative, no member of the population of interest are purposely excluded. • To determine the effectiveness of a new format test sent to a specific segment of customers residing on the database, for example, the direct marketer cannot restrict the sample to only those names living in New York. Doing so will yield results only reflective of New Yorkers

  8. Introduction -Why, What, How & When We Test (Continued) How we Test (Cont.) • The only exceptions to this rule should be: • Names also eliminated in roll-out such as DMA do not promotes, known frauds, credit risk accounts, etc. • Names recently test promoted for other marketing tests. • States or cities such as DC known to have strict promotional restrictions (especially important if the new test has not been fully reviewed by legal prior).

  9. Introduction -Why, What, How & When We Test (Continued) How we Test (Continued) In addition, the sample must also be drawn randomly or the test will yield biased and misleading results. • A random sample is one in which every member of the sample are equally likely to be chosen, ensuring a composition similar to that of the population. • To ensure a sample is randomly draw, many direct marketers utilize what is called “nth selects.” • To draw a random sample of 10,000 names from a database of 10,000,000 the direct marketer will begin by selecting one name on the database, choosing every 1,000th (10,000,000/10,000) name thereafter.

  10. Introduction -Why, What, How & When We Test (Continued) When we Test • Depending on the product or service being offered, you may be well advised to take into consideration seasonality. • For example, a travel product offered in October will yield a much lower response than one offered in April when families are planning their summer vacations. • Unfortunately we are not always able to test during the same time of year as the roll-out will occur. Therefore, you are advised to determine seasonality adjustments factors based on historical information. For example, a travel product offered in October is know to equal 80% of the response rate when offered in April. • With this information, you can make appropriate adjustments to your forecasts.

  11. Test Analysis Once tests are conducted, a marketing manager has two options available for analyzing the test results: • Hypothesis Tests • Confidence Intervals

  12. Test Analysis • Hypothesis Testing: This procedure will allow a marketing manager to determine if (a) the percent of favorable responses from a single test is significantly different from a certain value, or (b) if one test panel is significantly different from the control panel. • Confidence Intervals: If you are interested in knowing specifics regarding how much different your test result is from, for example, the control you will be required to construct a confidence interval. Confidence intervals will allow you to: • determine a range in which the response rate is likely to fall in roll-out based on the sample results, or • determine a range in which the difference in response rates between your test and control package truly lies based on the sample test results.

  13. Test Analysis/Confidence Intervals There are two types of confidence intervals that can be created. A confidence interval around a single test result You will use this formula when interest revolves around assessing the results of a single outside list, new product/service, or new house customer segment test. A confidence interval around the difference between two test results You will use this formula when interest revolves around assessing the difference between a your control and new format, offer or creative tests.

  14. Test Analysis/Confidence Intervals/A Single Test Response Rate You conduct a new outside list test to a sample of size 10,000 names and receive a response rate of 5.5%. Can you run to the bank with the 5.5% response rate?

  15. Test Analysis/Confidence Intervals/A Single Test Response Rate Absolutely not! Because you did not test the whole universe available, but only a sample, the response rate obtained is only an estimate. In fact, each time you conduct such a test you will get a different response rate.

  16. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Let’s say you ran not 1, but 10 different tests to the same outside list with the following results: Test # % Resp. Test # % Resp 1 5.1 6 4.7 2 4.8 7 5.2 3 5.7 8 4.5 4 4.3 9 4.9 5 5.5 10 5.3

  17. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Every time you test a list, you will get a different response rate. Some tests will yield results above the true response rate of the entire list and some below the response rate of the entire list.

  18. Test Analysis/Confidence Intervals/The Central Limit Theorem So, how can we possibly make any decisions with any level of confidence about what we can expect to receive in roll-out given the fact that there is the potential for so much variation in our testing results? The “Central Limit Theorem” is the answer!

  19. Test Analysis/Confidence Intervals/The Central Limit Theorem Let’s assume that instead of repeating the experiment 10 times we did it 1,000 times. And, assume you tallied the response rates received and created a histogram (bar chart) of the response rates using the Chart Wizard feature in ExcelTM.

  20. Test Analysis/Confidence Intervals/The Central Limit Theorem The tally of response rates might look as follows:

  21. Test Analysis/Confidence Intervals/The Central Limit Theorem And, if we produce the histogram from this data using the Chart Wizard feature of Excel, it will look as follows:

  22. Test Analysis/Confidence Intervals/The Central Limit Theorem Voila! The symmetric bell shaped or “normal” curve is revealed.

  23. Test Analysis/Confidence Intervals/The Central Limit Theorem • By way of The Central Limit Theorem, the following has been proven*: • The distributional shape of sample averages (or response rates as in our case) will closely resemble a bell shaped and symmetric normal distribution. • The average of these sample averages (or response rates) will approximate the true population average (or response rate). • And, the sample standard deviation** when divided by the number of observations in the sample will approximate the true population standard deviation (or spread). Note: This information is not found in the text book. • ________________________ • * These statements are only true for large sample sizes (to be revealed later) and becomes more true as the sample sizes approaches infinity. • ** The standard deviation is nothing more than an average of the deviations of each observation from their mean. Simply speaking, it is a measure of the spread of the data. The larger this number for a particular set of data, the more “spread out” the data values are.

  24. Test Analysis/Confidence Intervals/The Central Limit Theorem • For any set of data that is distributed normally (having a symmetric bell shaped curve), the following can be said: • 99.7% of the observations in the data set will lie within 3 standard deviations of the mean, • 99% within 2.575 standard deviations of the mean, • 95% within 1.96 standard deviations of the mean, • 90% within 1.645 standard deviations of the mean, and • 68% within 1 standard deviation of the mean.

  25. Test Analysis/Confidence Intervals/The Central Limit Theorem And, we have just derived the confidence interval formula based on The Central Limit Theorem.

  26. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • Constructing a confidence interval will allow you, the marketer, to assess the likely range in which the true response rate will lie based on your test. • To calculate a confidence interval around a single test response rate, the following information is required. • The sample response rate p obtained from the test • The sample size n of the test • The desired confidence level c And, n x p and n x (1 -p) must both be greater than or equal to 5. ^

  27. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) To determine the confidence level c, you must answer the following question: “How confident do I want to be that the interval I construct around my test response rate will contain the true response rate I can expect to achieve in roll-out?” Do you need to be 90% confident? 95%? 99%?

  28. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) The value chosen for c will guarantee, with the same level of probability, that the interval constructed around the test response rate will contain the true population proportion you can expect to receive in roll-out. It is recommended all confidence intervals be constructed with a 90% or better confidence level. Employing lower levels will yield more risk than you should be willing to take.

  29. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) For example, an 80% confidence level implies there will be an 80% probability the true response rate to be expected in roll-out will fall within the constructed interval bounds. But more importantly, it also implies there is a 20% probability (calculated as 1 - c) the true response rate to be expected in roll-out will not be continued in the constructed interval bounds. We call this value the “error rate” associated with the constructed confidence interval.

  30. ^ ^ ( p )( 1 - p )/n ^ ^ ( p )( 1 - p )/n Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) In simplistic terms, the confidence interval around your test response rate is constructed by adding and subtracting from it a multiple of the “sampling error.” The “multiple” will depend on the desired confidence level chosen. The formula for the lower bound of the confidence interval is: p - (z) And, the upper bound: p + (z) ^ The “sampling error” associated with the test response rate. Also called the standard deviation of the test. The “multiplier” which equals 1.645, 1.96 and 2.575 for a 90%, 95% and 99% confidence level respectively. ^

  31. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Consider the following example: Assume the marketing director at ACME Direct, a direct marketer of books, music, videos, home product catalogues and magazines, tested their core magazine title to a new outside list of names. The size of the test was 10,000. The test results of this new list yielded a response rate of 3.42%. To assess the potential for this new list in roll-out, the marketing director decides to construct a confidence interval. In particular, she needs to calculate an interval around this sample response rate such that she can be 95% confident the true response rate she can expect in a full blown roll-out will fall within the bounds.

  32. ^ ^ ( p )( 1 - p )/n Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) The marketing director will construct the lower bound of the 95% confidence interval as: p - (z) = .0342 - (1.96) (.0342)(1-.0342)/10,000 = .0342 - (1.96) (.0342)(.9658)/10,000 = .0342 - (1.96) .0000033 = .0342 - (1.96) (.0018165) = .0342 - .00356 = .03064 or 3.06% ^

  33. ^ ^ ( p )( 1 - p )/n Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) The marketing director will construct the upper bound of the 95% confidence interval as: p + (z) = .0342 + (1.96) (.0342)(1-.0342)/10,000 = .0342 + .00356 = .03776 or 3.78% ^

  34. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • The marketing director can be 95% confident, should she decide to roll-out with this new outside list, the response rate will not be: • less than 3.06%, or • more than 3.78%. • She can use the lower bound to determine the worse case scenario in terms of profitability by running a P&L calculation. • Based on her findings, she will decide whether or not to roll-out with this new outside list.

  35. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Setting the Confidence Level At what percent should you set the confidence level of your interval? In order to answer this question you must ask yourself the following question: How much risk am I willing to take in making an incorrect decision of assuming the true response rate falls within the constructed interval, when in reality, it falls outside of the interval?

  36. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • When you are constructing a confidence interval around a single test estimate to gauge a new list’s or new product’s potential, you will need to determine which of the following situations best fit your testing circumstances: • If the costs associated with a new list or new product are significantly higher in comparison to other lists or products currently being promoted, then there is a major risk associated with rolling-out with the new list or product if, in reality, it ends up performing badly. In other words, you really need to have a handle on the worse case scenario in term of response so set your confidence level high (95% or 99%). • And, by setting the confidence levels at a high rate (95% or 99%) you will minimize the chance of concluding the list or product test has met your requirements when, in reality, it did not.

  37. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • If the costs associated with a new list or new product are similar to other lists or products currently being promoted, then there is certainly less risk associated with rolling-out when, in reality, it ends up performing badly. As a result, you will set your confidence level at “industry standard” levels (90% or 95%) because the risk in making an incorrect decision is not as high, relatively speaking, as in the first scenario.

  38. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • If costs associated with a new list or new product are lower in comparison to other lists or products currently being promoted, then there is also less risk associated with rolling-out with the new list or product if, in reality, it ends up performing badly. Therefore, you will set your confidence level at “industry standard” levels (90% or 95%) since the risk in making an incorrect decision is also not as high as in the first scenario.

  39. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • Let’s assume for our ACME Direct example that this new list is more expensive than others currently being used for prospecting. In particular, this list costs $175 per thousand versus an average of $150 per thousand for lists currently being used. • What confidence level should the marketing director use?

  40. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Interpreting the Confidence Interval Once the confidence level has been determined and the confidence interval constructed, you are ready to interpret the results. Be forewarned, the interpretation of the interval is not “black or white.” Business experience and knowledge plays a major role in the final interpretation and decision made. A confidence interval will not provide you with a definitive answer to a business questions, but rather provide you with valid best and worse case scenarios to consider.

  41. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) For example, going back to our previous example the resulting 95% confidence interval was (3.06%, 3.78%). With 95% confidence, she knows the lowest this test list will respond in roll-out is 3.06% and the highest is 3.78%. In order to assess whether or not to roll-out with this new list, she will first examine if a slightly less aggressive confidence interval (90% in this case) reveals a different action be taken. If the lower bounds of both intervals reveal her “profit” criteria are met, then the answer is easy.

  42. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • If the two intervals are telling her conflicting information, she will need to do further analysis. In particular: • First, determine how close her “go/no-go” response rate level for the outside lists is to the lower bounds of both intervals constructed. • Next, consider the true upside and downside potential of a business decision by performing profit calculations using the upper and lower bounds of both confidence intervals. What are the worse and best case profit scenarios telling her?

  43. ^ ^ ( p )( 1 - p )/n Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Going back to our example, we determined a 95% confidence interval was the appropriate level, so let’s construct a slightly less aggressive interval at 90% and see what that is telling us. Lower bound: p - (z) = .0342 - (1.645) (.0342)(1-.0342)/10,000 = .0342 - (1.645) (.0018165) = .0342 - .00299 = .03121 or 3.12% ^

  44. ^ ^ ( p )( 1 - p )/n Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Upper bound: p + (z) = .0342 + (1.645) (.0342)(1-.0342)/10,000 = .0342 + (1.645) (.0018165) = .0342 + .00299 = .03719 or 3.72% ^

  45. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) • So, we now have the following two intervals: • 95% = (3.06%, 3.78%) • A slightly less aggressive interval at 90% = (3.12%, 3.72%) • Question: If we assume the minimum response rate the marketing director can accept for this list is 3.00% (which relates to a lose per paid sub of $25 or less), do you advise she roll-out with this new list to larger quantities? • Question: What if the minimum response rate of 3.10% was required? • Question: What if the minimum response rate of 3.35% was required?

  46. Test Analysis/Confidence Intervals/A Single Test Response Rate She can also run a P&L on the lower and upper bounds of both the 90% and 95% confidence intervals. For the chart below she assumed a projected payment rate adjusted a bit given experience. Question: If she is currently only pursing lists that cost no more than $22 per paid sub what should she do here? Question: What if she was willing to spend as much as $25 per paid sub, but no more? Question: What if she was willing to spend as much as $30 per paid sub, but no more?

  47. Test Analysis/Confidence Intervals/A Single Test Response Rate (Continued) Luckily, you do not need to use the complicated formula previously given to calculate confidence intervals. With the help of The Plan-alyzer, a software package created by Drake Direct, you can easily create confidence intervals around a single test response rate

  48. Test Analysis/Confidence Intervals/The Difference Between Two Test Response Rates When interest revolves around determining if one test has beaten another, you will be interested in examining the difference in response rates. ACME Direct conducts a new format test against the control format for their core magazine title based on samples of size 10,000 each. You receive a 5.85% response rate for the new format test and a 5.45% response rate for the control format. Can you go to the bank assuming the new format has beaten the control?

  49. Test Analysis/Confidence Intervals/The Difference Between Two Test Response Rates Absolutely not! Because both results are based on samples, each will have a certain amount of associated “sampling error.” The true difference is not .40% but something more or less than this percent. A confidence interval constructed around the difference between the two test response rates will allow you to determine the range in which the true difference actually lies by taking into account the amount or error associated with both tests.

  50. Test Analysis/Confidence Intervals/The Difference Between Two Test Response Rates (Continued) • To calculate a confidence interval around the difference between two test response rates, the following information is required. • The sample response rates p1 and p2 for both tests • The sample sizes n1 and n2 for both tests • The desired confidence level c And, n1 x p1, n1 x (1 -p1), n2 x p2 and n2 x (1 –p2), must all be greater than or equal to 5. ^ ^

More Related