ASIS&T 2005 START ConferenceManager    

Measuring Experienced Utility of Information Retrieval Systems: Experimental Approach

Irene Lopatovska

Sparking Synergies: Bringing Research and Practice Together @ ASIST '05 (ASIS&T 2005)
Westin Charlotte, Charlotte, North Carolina, October 28 - November 2, 2005


The poster presents the results of an experimental research project at the University of North Texas, Information Science Ph.D. program on a grant from the Texas Center for Digital Knowledge (TxCDK). The experiment was largely based on a naÔve methodology suggested by William Cooper in 1973 for applying utility in measuring effectiveness of Information Retrieval (IR) systems. The concepts of expected utility, experienced utility, instant utility, remembered utility, and predicted utility were borrowed from the fields of psychology (Schreiber & Kahneman, 2000) and behavioral economics (Kahneman, Wakker, & Sarin, 1997) and tested on an experimental system and individual systemsí outputs (e.g. abstracts, articles). The resulting analysis identifies the relationships between a) the subjective valuation of individual output and outputís contextual relevance to the participantís information need; b) the subjective valuations of individual outputs and the valuation of the overall system; c) the subjective valuations measured immediately after systemsí uses and after a period of time. Utility measure was selected as a central concept of the research. Utility refers to a measure of userís affective valuation that helps to explain choices made in the decision process. In information science an umbrella term relevance is often used to define concepts explaining usersí decisions to accept or reject information (Schamber, 1994), while utility is usually defined as usefulness of the information or search results (Su, 1998; Regazzi, 1988). Cooper (1973) defined utility as whatever the user finds to be of value about the system output: usefulness, entertainment or aesthetic value. Cooperís concept of utility was never applied in information sciences for measuring IR systems and their outcomes. However, utility as affective valuation measure was extensively tested in other social sciences. The discussed experiment involved 100 Business major undergraduate students. The participants were asked a question and given access to the experimental system to find an answer to a question. While conducting their search, participants were asked to rate their satisfaction with individual IR systemsí outputs and their satisfaction with the overall IR system performance. In addition, participants were asked an open question ďWhy did they rate the output/system the way they didĒ, identifying factors that impacted their evaluation. Two weeks later the same participants were asked different questions and were asked to find answers in the same experimental system. Repeating conditions validates the measure and allows the researcher to collect data on the remembered utility. Experimental data was analyzed using Regression, ANCOVA, and Structural Equation Modeling techniques. Application of interdisciplinary concepts of utility to information seeking situations is innovative and offers empirical methods for measuring subjective effectiveness of IR systems and their outputs and compares it to objective measures. The implications of applying utility as a measure of the IR systemsí effectiveness include better understanding of systemsí value to the users, leading to improvements in information systemsí design and pricing strategies; understanding of userís information behavior, including selection of the outputs to answer information need and formulation of perceptions regarding information systems; and understanding of information as a unique but measurable resource.

START Conference Manager (V2.49.6)