Primary Investigator

Email

Institution

Schober, Michael F.

schober@newschool.edu

New School University

Abstract
The objective of this research is to determine how best to design computer systems that collect survey data from users. In current practice, computer-administered surveys present questions or probes to all users in the same format, leaving the interpretation of those questions up to the user. This is done in the name of standardizing the materials—making sure all respondents are exposed to the same stimuli. But this practice is problematic because different users can interpret the same questions quite differently, which can lead to poor response accuracy under certain circumstances. Even if a survey system offers clarification for users, many of them may not recognize that they need it, or they may be unwilling to request it.

The studies in this project examine which sorts of interfaces to survey systems promote more uniform interpretations of questions—and thus higher quality data. The studies focus on desktop (windows, keyboard entry, and mouse clicking) and speech survey interviewing systems that government agencies are likely to implement in the near future, to gather the data for influential statistics like the unemployment rate or the Consumer Price Index. These surveys ask questions about verifiable facts and behaviors, rather than about opinions and attitudes. As such, they provide an excellent research opportunity: if the accuracy of users’ responses is verifiable, then objective evaluation of how different interface designs affect the uniformity of users’ comprehension and the ultimate data quality is possible. Because many government agencies have created official definitions for concepts in their survey questions—for example, defining what kinds of activities count as “work for pay”—it is possible to determine which interfaces promote question interpretations that match the official definitions.

The project consists of three sets of laboratory experiments using actual and simulated (Wizard of Oz) systems. The first set of studies examines response accuracy and user satisfaction for systems that give users clarification even if they haven’t asked for it, by monitoring their speed of responding and speech patterns. The second set of studies examines user response accuracy and satisfaction for interfaces that do or do not tailor the clarification in dialog. The third set of studies contrasts interfaces that require users to educate themselves about how the questions should be taken with interfaces that engage users in dialog to figure out the correct answer from the users’ perspective.

The purpose of the project is not to implement actual working systems, but rather, using the tools of experimental psychology, to determine what kinds of interfaces will be most effective, as determined by measuring response accuracy, duration of interactions, and user satisfaction. The findings will thus provide guidelines for future development of interfaces for collecting information from users, both for official statistics and for commercial purposes. More generally, the findings will contribute to the understanding of how design for systems that collect information should differ from design for systems that provide information to users.