Concerns about statistical power of these results

The most common question and concern regarding our first poll in September 2002 on the War on Terrorism was: With a sample size of only 150 how does the power of your results hold up against the major media polls? (which often have 800-1000 responses). A few people, instead of asking the question, have simply told us that our sample size is too small to be of any value.

Response: The focal point of Retro Poll's report and general methodology is an effort to look at correlations between misinformation people obtain from the mass media and their beliefs/viewpoints about proposed government policies. The association we have shown between having misinformation on Iraq's sponsorship of Al Qaeda terrorism on the one hand, and a willingness to go to war with Iraq, on the other, is very strong. Using standard statistical testing (the chi square test) we have shown that there is almost no chance (p<.001) that this association is a random or accidental event. In fact, this association is stronger than anything that the large random sample polls can say about their results. Health researchers and statisticians usually suggest a sample size of at least 100 to make this kind of study useful. Retro Poll 1 obviously surpasses that standard.

Comparing Retro Poll's association of two questions (factors) to the results of simple polls (whether major media polls or our own) is like comparing apples and oranges. The general public needs to understand the serious weakness of public opinion research in the U.S. Most public opinion polls (including ours) have to call and speak with about 4 people before one person agrees to participate in any survey. Since there is no way of knowing if the refusers have different or the same views as the accepters, there is no valid way to report that a poll really reflects the public viewpoint on anything. When polls report a margin of error (usually 3-5%, though our small sample size generated a statistic of 6-8%) they are, to put it gently, fibbing. What they should be saying is "if the people refusing to be polled were to have similar views to those polled, then the results reflect those of the general population with a margin of error of 3-5%". However, because the answer to the "if" is usually unknown, the actual potential error margin, in simple questioning, both in corporate media polls and in ours, is unknown but definitely larger than reported.

Retro Poll will continue to focus on the method of testing the association between background knowledge and viewpoint and these correlations will not require large numbers. We will however, try to get larger poll numbers in order to further strengthen the ability to say that results to individual questions reflect the public's views, even though we believe that this type of polling should not be trusted by the public, even when the sample size is large.

Return to Frequently Asked Questions