Theres a reason that Internet polls often bear disclaimers, noting that they are not to be taken as scientific studies. In this article, Ill explain some of the reasons for such cautionary labeling, using our recent SETI Institute/SPACE.com survey on attitudes toward extraterrestrial life to illustrate the points.

Though it would be tempting to dive right into the results of our survey, its important to be clear about some of the challenges facing the social scientist who would use the Internet to conduct surveys. Although we can describe many of the steps that social scientists go through in their work, its critical to recognize that the conclusions we can draw based on data collected over the Internet are unlikely to stand up to the rigorous scrutiny of conscientious researchers who rely on other ways to gather data, such as face-to-face interviews.

Nevertheless, as promised in the first article in this series, we can meet our goal to help people "gain a first-hand understanding of how social scientists use surveys to prepare for a SETI success," even if we cant be as confident in the specific findings as we might like. In short, while the final product of our web survey may still leave room for improvement using more controlled studies, the general process we use to collect and analyze the data will shed light on some of the tools that social scientists have at their disposal for understanding peoples reactions to detecting life beyond Earth.

One problem of using the Internet for surveys is that we have little control over who completes the questionnaires. True, there are some steps we might use to minimize the number of people who answer the survey more than oncesuch as keeping track of Internet addresses, which gives some (albeit imperfect) indication of whether a person has filled out the survey before.

But even in the best scenarios, Internet surveys can provide very skewed impressions of peoples attitudesin part, simply because its hard to generalize from Internet users to the general populace. If we discover that visitors to SPACE.com are overwhelmingly positive about the prospects of making contact with ET, its hard to know how closely that corresponds to the views of people who 1) dont use the Internet, and 2) dont have special interests in space activities. While we can get around some of these problems by posting surveys at multiple sites, in the end, the problem of unrepresentative sampling is only minimized, and never eliminated.

Perhaps the most jarring experience that many people taking our web survey had was encountering topics that seemed to have no bearing on extraterrestrial life. Why in the world, people might reasonably ask, would we want respondents to a web survey about SETI to tell us whether they agree or disagree with the following: "Most humans have eight arms and four legs"? Or for that matter, to respond to the statement "Apples, oranges, and bananas are examples of fruit." (These statements and others in this article refer to the first version of the survey that we posted. We changed the specific items included in the survey after the questionnaire had been up for a day and a half, so if you took the survey and dont recognize any of the items mentioned in this article, think about comparably "out of place" items in the version you completed.)

At first blush, it might seem that the correct answers should be obvious, and for researchers, not particularly interesting. But items such as these do provide researchers with valuable information. They help identify people who are really paying attention to the survey, and not just randomly answering, or even worse, attempting to sabotage the survey by providing nonsense responses. To deal with these problems, we included several items that have no direct bearing on beliefs about extraterrestrials, such as, "A triangle has three sides" and "Without exception, all cats feel very loving toward all dogs."

What is the value of identifying people who werent taking the survey seriously? It lets us simply remove their responses from our analysesor alternatively, to analyze their responses separately from the rest, letting us understand how these peoples responses differ from those of the more focused respondents.

In fact, the data that we gathered from over 3000 people during the first day and a half that the survey was posted on SPACE.com suggests that things arent quite so clear-cut. Some of the items intended to be obviously true or false were seen by some as ambiguous.

One way to track peoples reactions to some of these "odd" statements is to read the SETI Message Boards. Several people who filled out the survey commented on some of the items that seemed out of place, though some of these "checking" items seemed to cause more problems than others.

In fact, the section of the Message Board dealing with this survey starts out with the following message, posted by "crix":

Although we had intended that item as a test to see which respondents were paying attention, its complexity may have had another effect: it confused people. And in fact, more people made a mistake on that item than some of the other "checking" items. For example, 96 percent of the respondents either completely disagreed or moderately disagreed that "Most humans have eight arms and four legs." That is, only 4 percent of people got this item wrong.

Similarly, 95 percent of the people completing the survey agreed (either completely or moderately) that "Apples, oranges, and bananas are examples of fruit." In comparison, only 83 percent of the respondents got the "two plus two" item right. The same percentage of people (83 percent) made mistakes on the somewhat convoluted statement "Without exception, all cats feel very loving toward all dogs."

What do we learn from these different error rates? That some items may be better than others for identifying the haphazard respondent. SETI Message Board member "glynebwy" summarized this well, at least for some people completing the survey: "its not hard to filter out all responses that disagree a triangle doesnt have 3 sides for example. The 2 + 2 question though is ridiculous, it makes so little sense people m[a]y randomly click on anything for it." Our best bet for identifying hasty responders, it seems, would be to focus on peoples knowledge of how many limbs humans have and on how to categorize fruit.

Having highlighted some of the limitations of web-based surveys, we are set to move on to the next article, where we will discuss of some of our findings. In the process, we will come a bit closer to understanding some of the many ways that people may be affected if some day SETI discovers that we are not alone in the universe.