surveymasteronline

Online surveys and focus groups might solve the toughest problems in market research. But can Internet users really speak for everyone?

When Tee Ann Hunter is asked to participate in a survey, she usually does. "I don't mind market research, as long as I can see or hear the person I'm talking to," she says. She spends $50 on takeout food and $150 on groceries every week, plus about $200 a month on clothes for her three children, so her opinion is valuable to marketers. Yet she may never surf the Net, which could ultimately make her less visible to market researchers.

Hunter has a professional job and computers at work and home, but they aren't hooked up to the Internet or an online service. "The kids can go online at school, but I don't allow it at home because it's too hard to control. There's so much garbage out there," she says. "Myself, I have no need for it. I like to go out and do things. I don't understand people who 'surf' the Internet and send little messages back and forth. Maybe they don't have enough to do."

Meanwhile, millions of Americans discover the online world every year. About 23 percent of people aged 16 and older in the U.S. and Canada have used the Internet in the last month and have access today, according to the January 1997 wave of an ongoing survey by CommerceNet and Nielsen Media Research. In the fall of 1995, only 10 percent had used the Internet in the three months prior to the survey.

As the number of Internet users grows, their demographic profile becomes more similar to that of the average American. About 58 percent of current Internet users are men, compared with 66 percent in fall 1995. About 39 percent of users hold professional or managerial jobs, down from 50 percent in 1995. On the whole, though, Internet users are still an elite group. Fewer than 18 percent of all Americans aged 16 and older hold professional or managerial jobs, for example.

Despite their current bias toward the young, the male, and the affluent, Internet users are seen as a godsend in the troubled field of market research. About 17 percent of marketers surveyed who work in some of the 2,000 largest U.S. companies have used data from online surveys, according to an October 1996 study by the Council of American Survey Research Organizations (CASRO). Most of those who have used online surveys expect to use more of them in the future, and 42 percent of marketers who have not conducted research online expect to try it within five years. "As a researcher, I am agog at the potential for this new tool," says Nick Tortorello, a senior vice president at Roper Starch Worldwide, Inc. and CASRO committee chairman.

Believers in online market research say it is usually less expensive than a survey taken by phone, mail, or personal interview. It can deliver results from a large number of respondents literally overnight. It can find hard-to-reach groups, such as the owners of specific models of products, with relative ease. Almost every form of market research has now been tried online, from the simplest demographic surveys to the most complex focus groups and new-product research.

As firms rush to master this new marketing tool, the opinions of wired Americans are likely to affect more marketing decisions. But can Internet users really speak for everyone? And if they do, who is going to pay attention to people like Tee Ann Hunter?

WHY RESEARCHERS LOVE ONLINE

The quiet crisis in market research is that most Americans now refuse to be surveyed. About two-thirds of households have a telephone answering machine, and about half of householders sometimes use their answering machine to screen their calls, according to a 1995 study by CASRO. When they are contacted by telephone, about six in ten Americans refuse to answer survey questions. This problem is getting worse in a hurry. In a 1988 study, the refusal rate for telephone research was about four in ten. The situation is also grave for surveys taken face-to-face or by mail. The Census Bureau assumes that about four in ten U.S. households that receive a 2000 census form won't fill it out, even though they are legally required to do so.

Online research sidesteps the non-cooperation problem because all responses are voluntary and filled out by respondents at their leisure. There's no need to hire and train interviewers, and no chance that an interviewer's mistake will taint the results. No one's dinner is interrupted by an unwelcome phone call. In fact, the best times to collect online responses include late evenings and holidays. "We put up a study on Good Friday and had 2,400 completed forms on Monday morning," says Steve Cook, senior vice president for Greenfield Online Research Center in Westport, Connecticut. "People go home over the weekend, relax, and return their e-mail."

Greenfield Online has a "panel" of about 200,000 online users. Members fill out a short survey on their demographic characteristics, product use, and computer use; the company draws survey samples from this pool and offers participants cash or the chance to win a prize. "Our panel members are usually research virgins," says Cook. "They refuse other forms of research but participate in online research because it's not intrusive. They feel good about participating. We don't get a lot of pattern answers, which indicates that people take the time to think about the questions. And we get incredible open-ended comments."

One reason why marketers love online research is because they themselves are comfortable in the wired world. More than nine in ten marketers surveyed by CASRO were "very" or "somewhat" familiar with the Internet. Moreover, marketers report spending a lot of time--an average of ten hours a week--using online services for work-related purposes. "I'm in the business, but I don't have time to answer telephone surveys," says Maria Cheung, senior manager of market research for American Express. "But I will answer an online survey in a spare moment. I think it's still a novelty to many people."

WHY THE BOSS LOVES ONLINE

The main reasons for the explosion in online research are its speed and low cost. You can ask people what they would think of a price increase for your product on Friday and get data on Tuesday, before you have to make a decision on Wednesday. Also, you can do it without hiring, training, or monitoring interviewers. "Other kinds of research can't compete with that," says Nick Tortorello. "Online levels the playing field for smaller businesses that can't afford big research bills."

Online surveys could change the way many businesses operate. For example, television producers often solicit viewers' opinions of a pilot episode before they give the green light to a television series. Today it's common for 200 people to be recruited to view the taping of the pilot and fill out a survey, or for telephone interviewers to call households at random until they find a few hundred who happened to see the show. Now imagine that the show had an online address and viewers were offered an incentive to respond. It's not unreasonable to expect that thousands of people would respond the evening the show aired on their cable system, so producers could see how the pilot appealed to different groups defined by demographics, attitudes, or product use.

The new medium also makes it possible to find highly specific research targets without screening a large random sample. "One of our clients was trying traditional methods to find owners of a specific brand and model of a home-office product, purchased over a short span of years," says Steve Cook. The survey was a complicated, multi-stage process where respondents would first fill out a questionnaire and then install a computer disk with additional questions. "In six months of telephoning, they spent thousands of dollars and found just a few people," says Cook. "Within a few weeks of searching online, we found several hundred at a fraction of the cost."

Cook's firm also holds online versions of focus groups, where participants type out their opinions after being screened and invited into a "chat room" by a moderator. "They aren't a replacement for traditional groups, because you can't see body language and other nonverbal responses," he says. "But they are a lower-cost way to supplement traditional focus groups. You can mix different kinds of people--men and women, white-collar and blue-collar, older and younger--in ways that wouldn't work in person." The cost and speed advantages of online research are encouraging clients to push its boundaries. Some firms have tested advertising images by loading them on Web pages and collecting reactions. Cook's firm recently asked a panel to view pictures of products on a grocery shelf, then point to the ones that appealed to them most and "buy" them with a click of the mouse. In the near future, record companies might send selections of songs and videos to teenagers, in an online battle of the bands.

Online research almost sounds too good to be true, and some argue that it is. Online users are not a randomly chosen sample of an entire population, they say, so the results of online surveys cannot be used to draw conclusions about broad markets. In many cases, it's difficult to keep people from answering an online survey more than once. Clients also worry that competitors could steal peeks at their online research. In their haste to get results and save a few bucks, businesses that ignore the limits of online research could make expensive mistakes.

WHAT TO WATCH OUT FOR

All Internet surveys give an excellent profile of one target group: namely, the people who fill out the survey. For example, more than 1,000 people have filled out a short survey posted on the American Demographics Web site. Are they a good cross-section of our customers? No, because a large share of customers never go online. Are they a good cross-section of our Web site visitors? Probably not, because 70 percent of respondents say that it is their first visit to the site--and we have a lot of repeat visits.

Surveys of a small sample of any population accurately reflect the entire population only when the sample is "random." This means that any sample taken from the larger population must have the same probability as any other sample of being selected. To generate a random sample, you must first create a larger subset of the population that reflects, as closely as possible, the broad characteristics of the entire population. This larger subset is called the "sampling frame," and respondents are drawn from it at random. "Sampling bias" occurs when the sample is not perfectly random.

Some companies are trying to improve the situation. Burke, Inc. of Cincinnati, Ohio, offers an application that routes every nth Web site visitor to a survey section. While still not a true random sample, the program should provide a group that more closely resembles all the people who visit the site.

A few online surveys, such as those done by Greenfield, are drawn from a sampling frame. Greenfield's panel is large enough to provide some clients with good samples, especially if the target population is young, affluent, or male. But even then, clients must decide whether or not Internet users, as a group, have unique qualities and attitudes that add bias. "If you're trying to get the opinions of homemakers toward basic packaged goods, the Internet is not your vehicle," says Steve Cook. "We tell our clients that the results may not be projectible, and they need to be aware of the biases."

Another weakness of online surveys is the difficulty of verifying to whom you're talking. Stories abound of men who pretend to be women when they are online, or children who pretend to be adults. How do you tell? Some firms issue passwords to respondents once they are approved to participate in a survey, then require the password before the survey begins. "It's a problem now, but it will be solved soon," says Nick Tortorello. "Future computers will probably have the ability to read your fingerprint or your iris."

This "just-you-wait" optimism, so common to the Internet community, also describes marketers' views of online research. Marketers in the CASRO survey are aware that today's online surveys don't represent ordinary Americans. But most of them also say that online research in the future will be as reliable and accurate as mail, telephone, and in-person research.

It may also be possible to take advantage of online bias, because the attitudes of Internet users may predict the attitudes of the general public. Between 1993 and 1996, a sample of respondents to an online poll sponsored by Prodigy was drawn so that it was identical to the U.S. distribution for age, region, and sex. The online respondents' approval ratings for the President and Congress were compared with the general public's attitudes as measured by the Gallup Poll. "Prodigy tracked the general population's opinions almost exactly, and ran a little ahead of them," says Jan Werner, a Pittsfield, Massachusetts researcher who worked on the project.

Few samples of human populations escape from bias because some people are more likely than others to lie or refuse to cooperate. In the 1996 presidential election, for example, some studies found that Republicans were more likely than Democrats to refuse to answer exit polls. "Once the refusal rates in your sample become nonrandom, there is no way to project the results to the general population," says Werner. "Refusal rates are becoming so high that telephone surveys are using self-selected populations, just as online surveys do. Online may not be any worse.

"Researchers in the United States have a fetish for random samples, but it's a false issue," he says. "What's really important is establishing track records." In other words, if online surveys consistently and accurately predict the direction of your sales, they're good surveys for the purpose.

"Every method of surveying has a bias," says Tom Miller, director of Find/SVP's Emerging Technologies Research Group. "If you tell me exactly how a given online survey is collected, I don't think it's rocket science to figure out its bias." Find/SVP is considering a greater role for online surveys in its own research; it is now trying to devise new statistical weights and measures that would, as Miller puts it, "wring out as much sampling bias as possible." The cost and difficulty of taking a statistically rigorous survey continues to rise. Marketers need alternatives, and an online survey is often the best one available. "I need to reach a representative group of people who will answer my questions, and I'll do it however I can," says Cheung of American Express.

As the years go by, online surveys are likely to overtake older forms of market research. "The Internet is going to become mainstream communication," says Tortorello. "Everyone who hears about it likes the idea." Everyone, that is, except Tee Ann Hunter and an unknown number of Americans who agree with her. "E-mail, shmee-mail," she says. "If you want to talk to me, ask me in person."

--Brad Edmondson

TAKING IT FURTHER

An overview of the 1997 survey of Internet users by Nielsen and CommerceNet is $195. For more information, contact Patrick Corman at CommerceNet in Palo Alto, California; telephone (415) 326-9648; or online at http://www.commerce.net. Some results of the 1996 and 1995 surveys are available online at no charge. CASRO's "Survey Research Quality Guidelines" gives some information on how online surveys should be conducted. For more information on this or CASRO surveys, contact executive director Diane Bowers at 3 Upper Devon, Port Jefferson, NY 11777; telephone (516) 928-6954; fax (516) 928-6041. Burke, Inc. can be reached at 805 Central Avenue, Cincinnati, OH 45202; (800) 427-7057; Web site http://www.burke.com. Greenfield Online Research Center is part of The Greenfield Consulting Group, Inc., 274 Riverside Avenue, Westport, CT 06880; telephone (203) 221-0411; fax (203) 221-0791; e-mail dragon@greenfieldonline.com; Web page http://www.greenfieldonline.com. Jan Werner Data Processing is at 34 Williams Street, Pittsfield, MA 01201; telephone (413) 442-0416.

COPYRIGHT 1997 Copyright by Media Central Inc., A PRIMEDIA Company. All rights reserved.
COPYRIGHT 2004 Gale Group

posted by Claudia @ 7:25 AM