START Conference Manager    

ASIST 2012 Annual Meeting 
Baltimore, MD, October 26-30, 2012

Unreliable and Uncertain Annotators: Evaluating Rater Quality and Rating Difficulty in Online Annotation Activities
Organisciak Peter, Efron Miles, Fenlon Katrina and Megan Senseney

Sunday, 3:30pm


Gathering annotations from non-expert online raters is an attractive method for quickly completing large-scale annotation tasks, but the increased possibility of unreliable annotators and diminished work quality remains a cause for concern. In the context of information retrieval, where human-encoded relevance judgments underlie the evaluation of new systems and methods, the ability to quickly and reliably collect trustworthy annotations allows for quicker development and iteration of research. This study evaluates indicators of non-expert performance along three lines: temporality, experience, and agreement. In doing so, it finds that early user performance is a key indicator for future performance. Additionally, the time spent by raters familiarizing themselves with a new set of tasks is important for rater quality, as is long-term familiarity with a topic being rated. These findings may inform large-scale digital collections' use of non-expert raters for performing more purposive and affordable online annotation activities.