Web Exclusive

Coverage of ASIS 1997 Annual Meeting

Time as a Factor in the Evaluation of Information Quality

ASIS Annual Meeting Technical Session sponsored by SIGs HFIS/IAE, November 4, 1997

W. David Penniman, University of Tennessee, Knoxville: Loss of Quality in Emerging Information Systems
Donald H. Kraft, Louisiana State University: Possible Effects of Information Technology on the Peer Review Process: JASIS as a CASE History
Homer J. Hall, Rutgers University
Steve Hardin, Indiana State University

Session Abstract: The effects of better technology on quantity and quality of information are quite different. The rate of producing information can be increased while the time required for product evaluation improves less rapidly. This panel will discuss interactions between human skill and new technologies in case histories. (From the Final Program)

Session Report: Three speakers addressed "Time as a Factor in the Evaluation of Information Quality" at the ASIS Annual Meeting in Washington, DC. The session was sponsored by SIGs HFIS and IAE.

Moderator Homer Hall of Rutgers University was also the first speaker. He observed that "we get into trouble when the rate of information production exceeds the time it takes to evaluate it." In such cases, he said, information may be used without evaluation.

Hall gave his presentation in light of the 20th anniversary of SIG IAE -- Information Analysis and Evaluation -- which started life as SIG IAC -- Information Analysis Centers. Part of the reason for the transition, Hall said, was that IACs are places, while information analysis and evaluation are creative functions. He showed a slide of a pyramid presenting different levels of "information energy." At the base is the original information. Adding indexing and abstracting brings us to the bottom level of data storage base and retrieval. Adding compilation and reporting moves the process up to the middle level of the pyramid, information processing. Finally, asking "what does it mean?" brings us to the top of the pyramid, information analysis and evaluation. The higher up you go, the more experience you need to do the job.

In general, Hall said, the user doesn't know enough to ask the best questions about information; he or she is coming to the information analyst for guidance, and will rephrase the question based on what the analyst discovers. The user and the analyst must respect each other and work together. The analyst must be able to see things through the user's eyes. Creativity arises from interactions among different points of view.

Next, the editor of the Journal of the American Society for Information Science, Donald Kraft, used JASIS as a case history on the possible effects of information technology on the peer review process. JASIS, he said, needs to provide state-of-the- art information quickly enough for it to be useful. Partly to that end, the journal has gone from six issues a year to 10 to 12, and will publish 14 issues in 1998. Kraft hopes to reduce the year-long print queue.

Peer-reviewing a journal takes time, however. Reviewers will accept only so many articles, and some are very slow about returning articles they're evaluating. The bottom line, Kraft said, is that there are a lot of judgment calls to be made on the quality of a paper. "They simply have to be reflected upon and a decision made." He added he doesn't know how to shorten that process.

The new technologies have helped somewhat. But there are still a variety of formats, and operating systems. For that matter, not all authors have the same access. And often, sending the reviewer a paper on a disk results in the reviewer just printing the file and marking it up manually.

Kraft also considered some of the issues involved in putting JASIS on the web. The biggest question, he said, is copyright. Wiley, the JASIS publisher, is about as liberal as any publisher in its approach to copyright. But what about keeping a version of the paper on the web? What if the web version is different from the print version? What will happen with copyright as the years go by? These are all hot issues, Kraft said.

In addition, there has been a traditional perception among university tenure and promotion committees that the quality of electronic journals is not as high as that of print journals. Now that JASIS is electronic, will people still accept it as the "good old thing?" Kraft concluded that "information technology will affect us in lots of interesting ways, and we need to be careful to make sure the journals stay the quality you know and expect."

Next, W. David Penniman of the University of Tennessee at Knoxville talked about the loss of quality in emerging information systems. He cited Vannevar Bush's classic article "As We May Think" (Atlantic Monthly 176 (1945) pp. 101-108), to show many of the desirable attributes of a good information system were conceived more than 50 years ago. Penniman said he thought Bush would be especially pleased with the linkages between related topics widely available on the web.

Penniman also referred to user requirements as described in a 1959 article (Goodwin, Special Libraries, Nov. 1959, pp. 443- 446). Goodwin said users needed to get the information they desire, when they desire it, arranged in order of importance, with quality evaluations, with undesired information omitted, and knowing that negative results mean the information doesn't exist. He presented three definitions of "quality" for information:

  1. An absence of flaws or errors, often achieved be weeding out erroneous aspects at output stage;
  2. Meeting or exceeding user's expectations on a consistent basis;
  3. Fitness for use.

Penniman pointed out the things we associate with quality are not present on the web. He discussed web searches on "engineering" and "engineering libraries" in which the quality and the results structure depended on the search vehicle chosen. In addition, the reliability of the sources found -- "even to persist" -- is questionable.

The quality problem is compounded, he said, by the changing structure of information sources. Many of the standard sources found in the "good old days" no longer exist or have become commercial databases. Resources are diverging and the information scene is more chaotic.

Penniman also showed his audience highlights of a joint publisher/library study which indicated, in part, that publishers and libraries run risk of being excluded from the information dissemination process. If true, the information system could lose the quality control, information organization and other added values supplied by these entities. Selection and organization are not happening on the web!

What can be done? Penniman recommends returning to the basics of user requirements as described by Goodwin , then making use of Bush's vision. Systems need to be built, he said, around new capabilities that add value. At the same time, the older functions that also add value should be retained.

After questions from the audience, moderator Homer Hall summed up with a quote from Rudyard Kipling about 19th Century shipping innovations:

They asked me how I did it, and I gave 'em the Scripture text,
"You keep your light so shining - a little in front o' the next!"
They copied all they could follow, but they couldn't copy my mind,
And I left 'em sweating and stealing a year and a half behind.

- The "Mary Gloster," 1894