[Home Page] [Previous Page]
[Old Roundtable Archives] [New Roundtable]
Posted by Mike Curtis on June 30, 1998 at 12:17:22:
In Reply to: Re: Re: Re: Re: Re: Banner Ad / Online Survey Combinations posted by Randy Hill on June 30, 1998 at 10:25:40:
You make some good points, but let me borrow a famous example to help explain my point of view.
In the 1920s and 1930s Literary Digest conducted a presidential poll that was very famous. They had correctly predicted every presidential victor between 1920-1932. In fact, they predicted a Roosevelt victory in 1932 within 1% of the actual total.
In 1936 they conducted their poll as usual - a huge sample of 2.4 million respondents. Their prediction: Alf Landon 55%, Franklin Roosevelt 41%. Does anyone remember President Landon? Of course not. In actuality, Roosevelt won in a landslide - 61% to 37%.
So what happened? How could a sample of 2.4 million be so incredibly wrong? And how could it be so wrong after having been right for so many consecutive times prior to this?
As it turned out, two factors played a major role. One was nonresponse bias - although 2.4 million people responded, this was less than 1/4 of the total of 10 million people contacted. Republicans tended to respond in much higher numbers than Democrats, and Landon was a Republican.
Secondly, and more importantly, the poll used a very poorly designed sampling plan based on vehicle registrations and telephone numbers. Remember that this was at the height of the depression, and only the most well off people had phones and cars. These people also tended to vote Republican. The end result was a hugely biased sample that no one had anticipated because it had worked so well before. It resulted in a major embarrassment to Literary Digest, and led to their demise within two years.
At the same time, George Gallup used a sample of 3,000 respondents, but he also used scientific sampling methods. He not only correctly predicted the election result, but also predicted that the Literary Digest would incorrectly pick Landon before their poll was ever released.
So what's the point of all this? Just that my basic approach to research is to always keep debacles like this in mind. You can conduct poorly designed surveys and sometimes come up with the right answers. Other times you will come up with the wrong answers, and unfortunately, you have no statistical basis for deciding which is which.
Subject: Re: Re: Re: Re: Re: Re: Re: Banner Ad / Online Survey Combinations