 |
 |
 |
|
|
 |
|
|
Mike Curtis, Motoresearch Incorporated
Paper presented at:
Introduction A reality in the automotive industry is the incredible lead time required between concept initiation and bringing an actual vehicle to market. For example, the Chevrolet Venture minivan, which is just now coming to market, had its initial consumer research conducted in May, 1991. Delays in a product development program, even minor ones, can run into millions of dollars in costs. As a result, there is constant pressure to reduce the time requirements across all aspects of vehicle development. Automotive product research has not been immune to these pressures. Research suppliers to the automotive industry have been consistently pressed over the past several years to do it "better, faster and cheaper" than ever before. Subsequently, computerized data collection came into use in the early 1990s as a way of addressing these competitive pressures. Technological innovation in both hardware and software allowed Motoresearch and a few other market research companies to begin using computer-assisted self-administered interviews (CASI) in automotive consumer clinics at that time. Automotive Consumer Clinics Automotive consumer clinics have been around for more than three decades, since the early 1960s. They were a natural outgrowth of the automobile manufacturers’ interest in conducting consumer research on current and future vehicles. In order to conduct the research, manufacturers felt they had to show actual vehicles to respondents as test stimuli to provide a realistic environment for evaluation. Cost, timing and logistical considerations dictated that these events occur at centralized locations, with qualified respondents being invited to attend. Although computers have dramatically affected data collection and processing, clinics are not otherwise very different from their early incarnations. The target population for an automotive clinic is typically defined as the new car buying public. There is no readily available sample frame to identify new car intenders, so most often, new car owners are used as a proxy for intenders. Owners are identified through registration and sales data via companies such as R.L. Polk, TRW and Survey Sampling. A clinic is generally conducted in a single city due to cost, timing and logistical constraints. A city may be selected because its vehicle sales profile closely corresponds to national sales data, based on perceptions that it is leading-edge from a design perspective, or simply due to the fact that it has an adequate number of registrations for a particular set of vehicles to facilitate sample acquisition. Although external validity is a real concern, the limitations previously noted are bolstered by extensive historical data that suggest, for example, that a Ford Taurus buyer in Los Angeles is similar in expected ways to a Taurus buyer in Nashville. Geographical considerations may come into play with respect to characteristics such as front wheel versus rear wheel drive or sales penetration of leather interiors. In situations such as these, studies may be conducted in multiple locations if the issue is key to the research objectives. Clinics are very eclectic in nature; objectives differ dramatically. Subsequently, research designs vary across a wide range of alternatives. For example, sample sizes can range from as few as 50 people to as many as several thousand. Stimuli may be limited to a handful of vehicle photographs or include more than 90 vehicles on display. However, for purposes of clarification, a typical automotive clinic might have the following characteristics:
Before the proliferation of handheld computer technology, all data were collected using paper and pencil questionnaires. In fact, most companies today continue to utilize this method. However, a number of characteristics of automotive clinics lend themselves nicely to utilizing CASI:
Responding to pressures of "better, faster and cheaper," we began to seriously consider using CASI as an approach to data collection in 1991. Since that time we have conducted CASI interviews with well over 50,000 respondents in multiple languages, using both pen-based and keyboard-based computers. We began our CASI experiment when an outside computer software firm showed us a programmed questionnaire on a hand-held pen-based computer. At the time, our decision to go with a pen-based system was based on two factors. First, our gut feeling was that a pen-based system would be more intuitive and therefore easier to use for computer-phobic respondents. Second, notebook-sized computers were not yet being marketed and laptops were still too heavy to carry around and use as we envisioned. The pen-based system utilizing Gridpad computers and Ci3 software won by default. We used this system with modest success for over a year. As we began using CASI for our studies, a number of issues had to be addressed. The first problem we faced with the pen-based system was making our software functional. Ci3 was strictly DOS-based at the time, and DOS programs would not work with the pen-based system. The company providing the hardware to us, Advanced Data Research (ADR), also supplied an interpreter software program called Sidepad. The program worked as an interface between the DOS software and the pen-based hardware. The second problem we faced was the weight of the computer. Though a relatively light six pounds, we felt it was still too heavy for respondents to carry around for long periods of time. ADR provided a type of harness that would allow respondents to wear the computer, similar to the old-time cigarette trays. The harness was cumbersome to put on and interfered with respondents’ ability to evaluate ease of vehicle entry and exit, a key issue for automotive designers. However, given the difficulty of carrying the computers and the potential for dropping and damaging them, the tradeoff was made and harnesses were worn. One area that wasn’t a problem was respondents’ willingness to use the computers. It should be noted that none of the respondents knew ahead of time that they would be operating a computer during the course of participating in the research study. Though some were taken aback at first, the opptunity for first-timers to operate a computer seemed to be compelling. The novelty of using a pen-based system for otherwise experienced users proved to be a plus as well. People liked the system, and even apprehensive individuals were willing to give it a try. A guide attached the computer and harness to an individual and directed him or her to an orientation area. The person was shown a three minute video that explained how the computer worked and what to do if there were any problems. In general the computers turned out to be easy to operate and respondents quickly understood what we were asking them to do, but there were still a few problem areas. The biggest problem we faced turned out to be a combination of hardware inadequacy and screen design. The computer processor was equivalent to the early Intel 8086 microprocessor. Combined with the Sidepad interpreter software, the computer was very slow at accepting an answer. The screen was designed so that a long list of ratings would be shown one at a time, with the rating scale at the bottom of the screen. A respondent would touch a number on the scale and it would take about two seconds for the answer to register before bringing up the next question. Impatient respondents would tap the screen several times, thinking that the computer hadn’t accepted their answer. In the meantime a new, similar looking screen had appeared without the respondent realizing it. The result was that a respondent may have wanted to rate "interior comfort" a four on a five point scale, but in reality three consecutive ratings received a four, the second two unintentionally. We addressed this issue in two ways: first, we added a comment in the orientation process to alert respondents to the somewhat slow processing speed of the computers. Next, we made some changes to the screen appearance for each question. We started by blacking out the scale number selected by a respondent for 0.3 seconds before the screen changed. This was designed to let an individual know that the computer had accepted the answer. Next, we changed the line on the screen for which the rating appeared each time. For example, with the scale at the bottom of the screen we might show the first rating on line 10, the second one on line 11, and so on. The movement was designed to catch the respondent’s eye so similar looking statements such as "interior design" and "interior comfort" would clearly be differentiated. These changes reduced the frequency of occurrence, but did not entirely eliminate the problem. Other problems were difficult to solve as well. Nickel cadmium batteries were used in the computers, and they oftentimes did not last long enough for a respondent to complete the survey before requiring a change. The computer would have to be put into sleep mode, the old battery would be removed and a fresh battery installed, then the computer would be brought out of sleep mode. If these steps were not followed correctly, or if the bridge battery was weak, the computer would reboot and an interview would have to be restarted. As long as save commands were programmed after each question, no data were lost. However, saves added even more time to the overburdened processor, so this was a necessary but not ideal component of the programming process. There were additional issues with the hardware as well. The liquid crystal display (LCD) computer screens were not back-lit, making them hard to see in low light conditions such as when a respondent was seated in a car. The computers required frequent grid realignment to correctly pick up the pen’s electric signal. The signal itself was rather weak and therefore somewhat sensitive to dirt or fingerprint smudges on the screen. Even with these problems, however, we used these computers in virtually every domestic product clinic we conducted over a 14 month period from October 1991 through December 1992. In this time period we conducted approximately 30 automotive clinics and individuals completed over 10,000 surveys. Moving to a keyboard-based system The use of CASI had the effect of allowing clients to increase the complexity of the research designs. This further overburdened the pen-based system we were using. This, combined with the previously mentioned shortcomings of the hardware and the hardware-software interface, led us to consider other CASI options. At about this time manufacturers began marketing notebook and sub-notebook computers. These keyboard-based computers had a number of advantages over our then-current system. They were lightweight at about three pounds, they had back-lit screens, more RAM, greater data storage capacity and nickel hydride batteries. Furthermore, they ran on DOS which allowed us to use Ci3 directly. We had concerns regarding respondents’ ability to use a keyboard properly but felt we could overcome any problems. The benefits seemed to outweigh the problems so we purchased 35 Gateway 286 Handbook computers with which to conduct our studies. Over the course of the next year we added 25 more 286 Handbooks, followed by the purchase of an additional 12 486 Handbooks the following year. We have since conducted well over 50,000 CASI surveys using the Gateway/Ci3 system. Our first concern with the new CASI system, as suggested, was that non-computer users would have a harder time adapting to the computer than with the pen-based system. We considered making templates to leave only the required keys exposed or perhaps color-coding the keys to improve respondent understanding. The non-standard keyboard on these computers made these options somewhat impractical, so we decided to simply provide a detailed orientation video as we had with the pen-based units to see how respondents would react. The questionnaire was designed to require only a limited set of keys (number keys, arrow keys and the return key) to be used by the respondent. As added insurance we also decided to collect background data immediately after the orientation. This would allow respondents to learn how to use the computer while being proctored by an experienced guide. If any problems arose the guide would be able to address them. As it turns out, this process works extremely well, and we have changed very little since the first study. Most respondents complete the background section with no problems. Any problems that do arise are typically of a minor nature. It is a rare occurrence when an individual has problems operating the computer beyond this point. Other Considerations for Using Handheld CASI Handheld CASI has a number of advantages over paper and pencil. Many are similar to CATI advantages like randomization of lists and blocks of questions, prevention of missing data, allowance for complex skip patterns, etc. Additionally, the length and complexity of an automotive clinic further magnifies these advantages by allowing processes like recalling previous selections from an hour earlier in the interview and allowing split sampling assignments to different parts of the interview through randomization or quota procedures. Combinations of previous answers can easily be accessed for semi-structured interviews, e.g., "You rated Car X higher than cars Y and Z on exterior styling. Can you tell me what it was about the styling on Car X that made you rate it higher?" There are clear advantages to computer-assisted interviewing processes that need no further elaboration here. Extensive anecdotal evidence suggests that people can and will use the computers in a CASI environment. However, any decision to use CASI must take into account a number of issues related to questionnaire design, study flow, data management and hardware considerations. Questionnaire Design Issues Screen layout requires basic questionnaire design sense. Good paper questionnaire design criteria translate well into good CASI designs. Consistency is a key to avoiding confusion. The format for all similar questions should be the same; if the scale runs horizontally across the bottom of the screen for one question it should do the same for all. Instructions should be consistent with respect to screen location, font size and type, upper case and lower case usage and color, when applicable. Design issues must account for screen limitations. Most screens allow a 25 row by 80 column format. The layout depends on whether the screen is being formatted for a question or instruction. For question screens, avoid using multiple screens to display answer categories wherever possible. If there are too many answer categories for one screen, make it as simple as possible to page forward and backward through them. Making screens easy to read is the primary concern with instructions and other text-only formats. This would suggest use of multiple screens. One big advantage over paper is that the number of extra pages is inconsequential. There are no costs incurred for extra pages so fewer words can be spaced out across more pages, making instructions easier for the respondent to read. Another questionnaire design issue relates to the number of keystrokes a respondent needs to execute in order to answer a question. One of the most important considerations is whether to design the questionnaire to automatically trigger the next question when an answer category is selected or force the respondent to touch the "Enter" key after making the selection. This is a non-trivial matter in an automotive clinic where the latter choice could add 600 keystrokes to the respondent task. This issue must be traded off with potential measurement error via inadvertent response selection. What are the problems associated with making the process automatic? Consider this: On paper, when respondents circle an answer or write in a number, there is no doubt in their minds that the question has been answered. This is not always so on a computer screen. Imagine a screen with instructions at the top, a five-point rating scale at the bottom and an attribute to be rated such as "interior roominess" somewhere in the middle. The individual selects "4" as the response category. The next screen comes up with the same scale, same instructions and "interior comfort" as the item to be rated. If the respondent isn’t paying close attention, it appears that the computer didn’t accept the answer, and "4" is entered again. The respondent may do this three or four times before realizing that the question has, in fact, been changing. If the respondent lets us know, we can back the screens up and change answers. If not, we have garbage data for a few questions. To avoid this we typically program the screens to do two things: 1) blank the selected number out for 0.3 second, and 2) move the next rating down one line so the change is more obvious. On color screens, a change of color would work just as well. Wouldn’t it solve part of the problem if respondents could simply back up themselves? The answer to this question is yes and no. Yes it would probably eliminate a few wrongly entered responses, but in doing so it can create a whole new set of problems. Respondents have occasionally backed up more than 100 questions to change a rating on a vehicle or change a rank order preference. This can include backing through a logic tree and returning down a different path. If a counter is used in the program logic it can even result in the program bombing when the respondent tries to move forward again. To prevent these occurrences we typically allow backing up only within a narrow range of questions. Another aspect of the automotive clinic that affects the questionnaire design is handling open ended responses. This is typically done by asking respondents to complete their ratings and preference selections first, then directing them to an interviewing station. Here, an interviewer will take the computer and administer this part of the questionnaire. The answers are typed directly into the computer. When using pen-based computers a keyboard is plugged in before beginning the interview. Study Flow Issues Computer-assisted interviewing programs allow much greater flexibility than paper and pencil regarding randomization procedures. However, a number of complexities have to be overcome logistically in an automotive clinic that may not be faced with a CATI interview. For example, it may be methodologically ideal to randomly assign the order that respondents view study vehicles, but logistically it isn’t feasible to have people crisscrossing a 30,000 square foot facility searching for one particular vehicle out of 20. Instead, the vehicles are labeled alphabetically in a serpentine fashion. The computer randomly will assign a starting point, then randomly assign a direction, either up or down the alphabet. While this difference may seem minor, it oftentimes results in hundreds of additional lines of programming code compared to pure randomization procedures. Another interesting problem has been figuring out how to properly pace respondents through the study. A mean of 2 ½ hours may include a rather broad standard deviation among individuals, with some taking as long as 3 ½ to 4 hours to complete the study. A paper questionnaire inherently gives feedback as to the pace; if half of the pages have been completed a respondent can reasonably assume that they are halfway through the study. If too much time has been used getting to this point the pace can be increased. Unfortunately, the computer doesn’t provide this feedback automatically. Typically a respondent doesn’t know whether there are two questions left or 200 left. Therefore, the fact that the interview has taken two hours to this point is relatively meaningless. We have tried to overcome this problem in a few different ways, with varying success. First, we tried to show respondents the overall study flow in an orientation video. Here we described each task in detail along with the expected amount of time each task would take. The purpose was to help each respondent allocate an appropriate amount of time to the task at hand in order to maintain an appropriate pace. We have found, however, that too much information has had the same effect as giving them no information, except that we wasted an additional ten minutes of their time. The second attempt involved programming a flow chart into the questionnaire. After each subsection of the survey a flow chart appeared on the screen. It would indicate how far along the way the respondent was and how much was left to do. Additionally, it would show a clock to indicate elapsed time and compare it to the elapsed time of an "average" respondent. This would let the respondent know whether the pace was adequate or whether it needed to be increased. This process actually works quite well. Unfortunately it requires significant programming and the typical project time constraints and last minute changes preclude us from using this process very often. The newest version of Ci3 for Windows offers a progress bar as part of the program code to allow respondents to see the percentage of the questionnaire that they have completed up to that point. Although we haven’t used it yet, we will test it in the near future. Our third effort was to go back to a very simple orientation process where we lay out the tasks in very generic terms. We have even gone so far as to tell respondents that the time required to complete the study was based on the assumption of 15 seconds per rating scale question. This worked as a stopgap measure in a recent study where we underestimated the average completion time, but we have concerns that some people will focus more on meeting time constraints than giving us appropriate answers. Maintaining control over the flow of respondents through the study is also a key concern that affects study design. At any given time we may have dozens of people participating in one of our clinics. The nature of the scheduling, pacing and flow results in people being at all different points in the process at any given time. There is no easy way to know by looking at a person that he or she is at the correct point in the survey process, or even in the correct room, for that matter. Therefore, we hedge our bets a little bit on the instruction screens. The screen may, for example, instruct the respondent to "Proceed to Display 2 and see a guide." Here the guide will orient the respondent to the next task. To ensure that the respondent actually goes to see the guide, a secret lockout key prevents a respondent from advancing the screen beyond the current instructions. Once they see a guide, the guide can verify that they are in the correct room, give them an orientation to the task, then touch the secret key (e.g., "z") to advance the computer and allow the respondent to continue the survey. Data Management Issues One data management tradeoff to consider is the frequency with which data are saved as a respondent moves through the questionnaire. The tradeoff becomes the risk of lost data versus bogging the system down and reducing battery life with frequent access of the hard drive. The two extremes are obviously to save after each question or only after the entire questionnaire has been completed. The problem with saving after each question is the lag time required before advancing to the next screen. This is particularly true with slower 80286 computers. Again, with 600 or so keystrokes in an interview, this is not a trivial issue. It also tends to confuse the respondent; an answer has been entered yet the computer doesn’t seem to be advancing. The other side of the coin is to save only at the end of the interview. Here you run the risk of lost data if a respondent’s computer crashes along the way. A crash requires a restart, which will return the questionnaire to the last saved screen. With no saves the computer will simply "error out" and indicate that no such interview even exists. A compromise solution we typically use is to save at logical points in the survey such as at the end of a short list of ratings. If a list of ratings is long, say 20 or so, we typically will save after every fifth one. The worst case then requires a respondent to reenter up to four rating scale responses if the system crashes just before a save. Open ended responses are saved after each one since these are interviewer-administered anyway. As computer processing speeds continue to improve, this issue will eventually become moot, since no penalty will be incurred for saving after every screen. Another aspect of data management is how to handle storage and compilation. One alternative is to download and compile each file into a centralized database after each completed interview. This may be the safest procedure, but the downside is that it is time consuming and requires a full time data processing staff member to handle this single task during a study. The opposite choice is to collect the files on an individual computer and download all of them at once, either nightly or at the end of the study. Here the risk is losing 15-20 interviews if a hard drive becomes damaged at some point. Our process, which has worked very well, falls somewhere in-between. We download data files onto a floppy disk after each respondent completes the interview. This process takes less than a minute. At day’s end we take all disks and download them into a central database, where the data are then compiled. If any of the individual files are corrupt at this point we still have the original version on the handheld computer. This file is again downloaded and recompiled. Hardware Considerations Automotive clinics necessitate specific computer hardware characteristics. First and foremost, computers must be very light. Even a six pound computer, which is lightweight by most standards, is too heavy for respondents to comfortably carry around with them. We have found that computers in the three to four pound range are reasonable to use. However, even these become heavy if respondents have no opportunity to occasionally set them down. Battery life is a critical concern. The very nature of the study design precludes plugging the computer into a wall outlet in most circumstances. The length of the typical clinic may even exceed the useful battery life in some computers. This requires a battery change for most respondents during the course of the interview. This is usually a straightforward process, but if the internal backup battery is dead or the procedure is executed incorrectly, the computer will reboot at this point. Though it is possible to prevent data loss in these situations, it still takes a few additional minutes to reboot and restart the respondent interview. Computers have evolved through various battery types: original handheld computers work with nickel cadmium batteries whose deficiencies include a relatively short charge state and a "memory" problem that could prevent them from properly recharging over time. This memory problem occurs as a result of not completely running the battery down between charges, thereby causing the battery to recharge only partially. This makes computers with these batteries sub-optimal for handheld CASI usage in a clinic environment. Nickel hydride batteries last a little longer and recharge quicker than the nickel cadmium variety and can still be found in portable computer applications. Lithium ion batteries seem to be the norm in many new handheld computer applications, and are the most expensive as well. Many other things besides battery type can affect a battery’s longevity, including screen type, hard drive access frequency, microprocessor type, etc. Most handheld computers offer a battery saver mode to help maximize battery life between charges. Handheld computers also require easy to view screens, especially in poor lighting conditions such as inside vehicles. Back-lighting typically works well. A non back-lit liquid crystal display is very difficult to see in low-light conditions. Operational Cost Considerations When considering the cost of using handheld CASI versus paper and pencil, frequently left unaccounted for are the repair costs and depreciation associated with the computers. Respondents are hard on computers. They drop them, spill coffee on them and generally treat them poorly. Hard drives and screens are the two items most frequently damaged. A typical repair bill on one of our damaged computers is about $350. Our four year old computers have a fairly high mortality rate; currently it is common for 10-15% of the computers to require repairs following a study. Another concern is that computer technology gets old very quickly, with old models being replaced by new ones in rapid succession. For example, as previously noted, we purchased 60 80286 sub-notebook computers. Six months later when we went back to buy more we purchased their upgraded 80486 computers at a lower cost. Finally, when we went back to purchase more computers a year later, no version of the sub-notebooks was even available. Problems finding replacement parts and batteries for these computers are becoming more prevalent. This cycle of constantly improving hardware and software has ultimately led us full circle. We have just leased our next generation of handheld computers for CASI use; pen-based color screen computers with 486-100mh processors running Windows 95 operating systems. The warranty lasts as long as the lease does, so we’re hopeful that repair costs can be contained. We are further hopeful that the lease process will keep us current with future improvements to the technology. Summary Handheld CASI has been an effective tool for us in an automotive clinic environment. Perhaps first and foremost, the important point to be made is that respondents willingly use the computers and can be quickly trained to do so. Our initial concerns have proven unfounded. Even computer-phobic respondents are able to quickly learn to use the computer, whether pen-based or keyboard-based. Most enjoy the process and many first time users complete the study with a sense of accomplishment. Only one person in five years has refused to participate after finding out he had to use a computer. His response was, "I don’t use computers, period." Complex skip patterns and randomization procedures can be easily incorporated into questionnaire designs. Data integrity and lack of data entry requirements result in a huge data processing benefit after fieldwork is completed. New Windows-based hardware holds the promise of high quality graphics display, sound and even video. Another key benefit is the ease with which these computers can be used in multiple languages. All basic programming can be completed in English, then text that appears on the screen can be translated into any number of languages. We have conducted studies in French, German, Italian, Spanish and Portuguese using handheld CASI computers. Japan and several other Pacific Rim nations are on the horizon as well. With all the benefits, however, there are still many unanswered questions regarding the use of handheld CASI. For example, do respondents use scales differently on computer than they do with paper and pencil? Do multiple response questions elicit different numbers of items being selected depending on method? Are issues unrelated to social desirability affected? Differences in demographic characteristics have been shown to exist between respondents who elect to use CASI and those who don’t (Couper and Rowe, 1996). Are these characteristics also predictive of differences in response between the two methods? In other words, might respondents who are computer literate give answers consistent with paper and pencil, where others may not? Even within a CASI environment, differences have been found between CASI and audio CASI (Tourangeau and Smith, 1996). Do pen-based and keyboard-based formats elicit differences in response as well? Some research suggests that it may depend on the type of answer requested, i.e., closed-ended, numeric or alphanumeric (Couper and Groves, 1992). Do completion times vary between CASI and paper and pencil studies? Anecdotal evidence in automotive clinics suggests the completion times are similar, but what about in shorter studies where learning curves are likely to have a greater impact? Although these and other questions will certainly have to be addressed, I am very upbeat about the future of handheld CASI. Hardware and software will continue to improve. Computers will proliferate throughout all aspects of our lives, and society as a whole will become much more comfortable with their use. In the near future, computerized questionnaires will become nearly as common as paper and pencil ones were just a few short years ago. References Couper, M.P. and R.M. Groves. 1992. "Interviewer Reactions to Alternative Hardware for Computer-Assisted Personal Interviewing." Journal of Official Statistics 8:201-210. Couper, M.P. and B. Rowe. 1996. "Evaluation of a Computer-Assisted Self-Interview (CASI) Component in a Computer-Assisted Personal Interview Survey." Public Opinion Quarterly 60:89-105. Tourangeau R. and T.W. Smith. 1996. "Asking Sensitive Questions: The Impact of Data Collection Mode, Question Format, and Question Context." Public Opinion Quarterly 60:275-304.
©Copyright 2004 by Venture Data, LLC. All Rights Reserved. Contact Scott Spain with Web site questions and feedback. Site Sponsors: W3 Surveys and Venture Data
|
|
