START Conference Manager    

New Measures for the Evaluation of Interactive Information Retrieval Systems: Normalized Task Completion Time and Normalized User Effectiveness

P. Bryan Heidorn and Xiao Hu

(Submission #144)


User satisfaction, though difficult to measure, is the ultimate goal of Information Retrieval (IR) systems. In recent years, as Interactive Information Retrieval (IIR) systems have become increasingly popular, learnability also has become critical in evaluating IIR systems. However, existing measures in IR evaluation are not particularly suitable for gauging user satisfaction and ease of learning. In this paper, we propose two new measures, the Learning Efficiency Ratio (LER) and the Modified Learning Efficiency Ratio (MLER), to evaluate IR systems. While these two measures focus on evaluating the learnability of IR systems, we also investigate their relationships with user satisfaction and retrieval effectiveness. A user study was conducted examining whether the two measures were significantly correlated to the user satisfaction and effectiveness of a given IR system. The learning curves described by LER, MLER, and the task completion time were also studied and compared. The results show that both LER and MLER are strong indicators of user satisfaction, and both are better measures than the task completion time in measuring the learnability of a given IR system.


Program Track:  Posters and Demos
Submission Type:  Poster

START Conference Manager (V2.56.8 - Rev. 1261)