[Home Page] [Previous Page]
[Old Roundtable Archives] [New Roundtable]

Re: Questions, please help

[ Follow Ups ] [ Marketing Research Roundtable ]

Posted by John Taylor on February 02, 2000 at 12:35:04:

In Reply to: Questions, please help posted by karin on February 01, 2000 at 18:12:02:

I assume your universe, the total group about whom you would like to make some statements, is considerably larger than 1500. Lets say 10,000 for sake of argument. If you surveyed the randomly selected 1500 and got responses from ALL of them, you would apply a "margin of error" or confidence interval to the results to reflect the magnitude of the Sampling Error: the possible error that results when you collect data from a sample rather than the complete universe of people. So you still have this source of error to deal with.
On top of that, you have potential Non-Response Bias: the difference between the answers of the 1500 and the 400 who elected to respond. You have no way to quantify this possible source of error at this point. (There was a good series of posts on non-response error within the past two months.)

Your 27% response rate sounds quite good for a typical mail survey. This assumes your audience does not have a particular connection with the topic, and you did not provide a monster incentive. So, it sounds like you did a reasonably good job on the survey execution.

Nonetheless, those 27% were self selected. Given the controversial topic it seems quite possible that those who responded have different behaviors and opinions from the 1100 who did not respond. One possibility is to go back to (mail or telephone) a small sample of the 1100. If you get answers from these "non-respondents" that are different from your first 400 then that is a sign of significant non-response bias. Conversely, if the answers of these "non-respondents" are essentially the same as the 400, you would conclude the self selection process did not have to do with the subject matter.

In many surveys, the analyst assumes the later and simply calculates the margin of error as though he/she had selected a sample of 400 from the 10,000, in effect ignoring the non-response issue.

I prefer to inform those who would use the results about the risk of non-response error, and let them decide how much weight, or creditability, to give the results. The earlier series on non-response error includes a useful way of quantifying how far "off" your answers might be because of non-response.