ASIS&T 2006 - Dispelling The Myths Behind Straight Citation Counts

Dispelling The Myths Behind Straight Citation Counts

Dangzhi Zhao

ASIS&T Annual Meeting - 2006 (ASIS&T 2006)
Austin, Texas, November 3-9, 2006

Abstract

Counting citations received by scholars has long been used as a way to evaluate the impact of scholars. It has been noted theoretically and supported by small-scale studies that different citation counting methods can produce very different author rankings and that fractional counts are most preferred among the three commonly discussed methods: straight, complete and fractional counts. However, in practice of citation analysis studies, straight counting is still the most frequently used method. We examined the reasons and arguments for this common practice in the present paper through a full-scale comparative citation analysis study using all of the three citation counting methods just mentioned.

We took advantage of data and tools for citation analysis studies increasingly available on the Web and compared rankings of cited authors in the XML research field by the three different citation counting methods. Results clearly showed that the number of top-ranked authors shared between author rankings resulting from different counting methods was quite small, and that rankings by complete and fractional counts were highly correlated but were significantly different from that by straight counts. An analysis of the shifting patterns of authors between different rankings indicated that fractional counts are least biased among the three methods compared because the assumption underlying straight counts, i.e. that the first author of a paper contributes solely or significantly more to the paper than the other authors, frequently does not stand true, and complete counts unfairly favor authors who are members of large-group collaborations.

We thus provide strong empirical evidence for the bias associated with straight counts, and for the lack of significant correlations between author rankings by straight counts and other counting methods. We also demonstrate the capability of emerging tools such as CiteSeer in supporting various counting methods, which contests a major justification for the continued use of straight counts in citation analysis studies – the prohibitive costs involved in counting non-first cited authors using the ISI databases. Although it may require more technical skill and effort to work with them, these tools reduce the costs significantly. More importantly, these tools allow us to go beyond what the ISI databases have supported to explore more sophisticated citation counting methods for more balanced and less biased evaluations of scholarly impact. This may in turn contribute to the emergence of a more effective scholarly communication system in which scholarly work is evaluated based on how many users it has reached in all rather than on how many users it has reached who have published in journals indexed by certain databases such as the ISI databases.

START Conference Manager (V2.52.6)

Maintainer: rrgerber@softconf.com