[Home Page] [Previous Page]
[Old Roundtable Archives] [New Roundtable]
Posted by Sherman Whipple on February 07, 2000 at 15:30:30:
This whole business with on-line research is starting to drive me up the wall. A couple of years ago, when I had only audited and validated a handful of studies using on-line methods, I felt that on-line research held great promise for many of the research disciplines. At the same time, I was hesitant to include on-line techniques in the curriculums we developed, because I did not know enough and believed that it would be irresponsible to recommend any method which might produce misleading data.
As of today, I have now performed audits including over a hundred studies using on-line methods, and evaluated them in the context of both empirical data (ie. actual sales, leads, quotes, information requests) and in comparison to findings (ie awareness, intent, penetration) from traditional methods (ie. CATI, mail-out, panel, intercept, etc.). We have even had several occasions to evaluate data from identical instruments using both on-line collection versus traditional methods.
What I have found is that of the on-line methods used in the base of research I have audited, only those which fall into the panel method have produced data which can be shown to be relevant to markets or populations (either relative or projectable with weighting). This includes recruited/screened consumer panel, expert panel, Delphi panel and, even "on-line focus groups" or what I would call an ideation panel.
Because we see that on-line panel methods work, and there are a number of providers (ie. Harris-Black) with experience in this area, we are now recommending on-line panels to our clients.
I have also found that on-line feedback mechanisms (ie. site surveys) may also have merit, but only in the context of the site itself or as it may relate to on-line deliverables (software, support, catalogs, information, specifications, etc.) but that this data is not relevant outside of the on-line environment or the specific "browser base". Where we have market control data, we have found that the on-line response is skewed by around 300% from the market as a whole, and is often "squeaky wheel" or inverse to the point that what customers respond on-line is the opposite of what they actually will do.
Because we see that the data from site surveys so far appears to produce erroneous findings in relation to a business' market or real world position, we have been strongly recommending against on-line surveys for any sort of marketing or market research, with the exception of feedback mechanisms for the site itself. We reenforce this with the caveat that the client must not even think trying to apply this to his/her marketing decisions.
Now this is not a matter that I have closed my mind to on-line surveys, we have found on-line methods that work, rather, we feel that we are recommending the best usage of the Internet, and, with the exception of those specific uses, noted above, cannot in good conscience recommend any other approaches. The problem is that quite a few of the interests with whom we are currently working do not agree. Several are demanding that we specify on-line primary survey research to be used for marketing decisions by non-on-line companies.
So let me throw this up for discussion: Aside from panels does anyone know of a primary approach using the Internet that can produce projectable findings to the real world (US domestic)? Is anyone aware of some new direction or progress in this area?
Subject: