ASIS&T 2013 Annual Meeting 
Montrťal, Quťbec, Canada | November 1-5, 2013

Simulated vs. Real Work Task Situations: How Different Could They Make?

Yuelin Li, Nankai University
Die Hu, Nankai University

Monday, 1:30pm


Task design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and simulated work task situation design are used most widely, especially the latter. This paper attempts to examine the possible differences between simulated and real work task situations in a digital library evaluation, with respect to task characteristics, the influence of the tasks on usersí interactive information search behavior, and interaction performance. An experiment was conducted. One simulated work task situation was assigned to forty-two participants and they were also asked to bring one real work task to the experiment. A set of questionnaires, including an entry questionnaire, a pre- and a post-search questionnaire were administered to the participants. An exit interview was conducted before the experiment ended. The results indicate that the simulated and real work task situations are significantly different in some sub-facets of task, but not all examined in this study. However, the influence of the two tasks on usersí interactive information search behavior and interaction performance is not significant, though the sub-facets of task of real and simulated work task situations are correlated to interaction performance differently. Therefore, the simulated work task situations could be a supplement of real work task situations in IIR evaluation if well designed. Also, the study found that for some participants, they assessed the simulated one in fact easier, but the real one in fact harder after the search. Moreover, it should be taken into account to control some critical sub-facets of task when designing simulated work task situations. The paper concludes with suggestions on designing simulated work situations in IIR research and future studies.