Suppose that you were to read the following news story: "Researchers compared a new drug to a placebo for treating high blood pressure, and it seemed to work. But the researchers were concerned because they found that significantly more people got headaches when taking the new drug than when taking the placebo. Headaches were the only problem out of the 20 possible side effects the researchers tested." Do you think the researchers are justified in thinking the new drug would cause more headaches in the population than the placebo would? Explain.
A. They are justified, because they found a significant difference in the populations which experienced headaches. In other words, a statistically significant result always indicates an important relationship.
B. They are justified, because they likely took all necessary precautions to assure their testing methods yielded credible results. In other words, the observed relationship was correct, since the researchers were likely cautious about their testing methods.
C. Maybe they are not justified, because they likely used too many confidence intervals/hypothesis tests to determine if the drug caused headaches. Some information may have been ignored in the process. In other words, it is possible that the observed relationship was a false positive and occurred because of multiple comparisons.
D. Maybe they are not justified, because they only found a statistically significant difference in the populations which experienced headaches. A statistically significant relationship between variables does not necessarily indicate an important relationship.
E. Maybe they are not justified, because 20 side effects were tested, so one or a few could appear to be a problem (statistically significant) just by chance even if none of them were a problem. In other words, it is possible that the observed relationship was a false positive and occurred because of multiple testing.