Partial Relevance Judgments During Interactive Information Retrieval: An Exploratory Study

 

Amanda Spink, Howard Greisdorf

University of North Texas

 


Abstract

 

Usersí relevance judgments are central to both the systems and user-oriented approaches to information retrieval (IR) systems research and development. A basic assumption of IR and relevance research has been that users always require the most "highly" relevant items when submitting a query to an IR system. This study examines users conducting their initial online search on a particular information problem. Findings from three separate studies of relevance judgments by 44 initial search users were examined, including two studies of 13 end-users and a study of 18 users engaged in mediated online searches. Results show that the number of items judged "partially" relevant (on the scale: relevant, partially relevant or not relevant) was found to positively correlate with changes in users; (1) information problem definition, (2) personal knowledge due to the search nitration, (3) criteria for making relevance judgments. Items judged "highly" relevant were not correlated with these factors, and also (4) search intermediaryís perceptions that a userís question and information problem s changed ruing the mediates search interaction. Results of these three studies suggest that: (1) partial relevance judgments play an important role for users in the early stages of seeking information on a particular information problem, and (2) a relationship exists between partially relevant items retrieved and changes in the users information problem or question during an information seeking process. Findings also suggest that "highly" relevant items may not be the only items useful at the early stages of usersí information seeking processes. Implications for the development of IR systems and relevance research are also examined.