B  U  L  L  E  T  I  N

of the American Society for Information Science

Go to
 Bulletin Index

Volume 25, No. 4

bookstore2Go to the ASIS Bookstore

April / May 1999




Expanding Our Knowledge of Evaluating Networked Information Services and Resources:
Prelude to the Mid-Year Meeting

by Charles R. McClure, John Carlo Bertot and Carol A. Hert

The 1999 ASIS Mid-Year Meeting (MYM) in Pasadena (May 24-26) presents an outstanding opportunity to expand our knowledge of evaluating networked information services and resources. As co-chairs of the meeting, we would like to provide a preview of some of the important and timely issues that the Society will be discussing there.

Access to and use of networked information resources and services over the Internet continue to explode. Traditional uses, such as e-mail, Web searching and uploading/downloading files, have continued to grow exponentially at the same time that new and innovative interactive services and applications such as moving from "Webs" to "portals" are also appearing and taking hold at an astonishing rate. Commercial applications and uses of the network also are growing. As the "Digital Economy" expands, multiple impacts on society will occur.

The label networked information services and resources designates many complicated interactions among technology, policy, organizations and people that require ongoing thought and clarification. Network-based resources and services include those electronic information resources and/or services that users access from their homes, workplaces, libraries or other locations via public, commercial, statewide or national networks. Some examples are Web sites providing public and commercial information, as well as library-licensed databases (e.g., Infotrac, SearchBank, EbscoHost). Examples of electronic networked services include e-mail, listservs, chat, online reference/assistance, downloading information or forms, interactive video and "push" technologies that provide users with a range of customized information and services on a regular basis.

This complicated web of technology, services and applications, users and organizations and the need to assess its implications and impact led to the planning of this Mid-Year Meeting. The call for participation, which can be viewed at http://www.asis.org/Conferences/my99call.html
outlines the goals and topics that the MYM papers and panels will address. Notions of evaluation and use of networked information services and resources include the following:

  • use
  • measurement
  • theory
  • applications
  • education
  • social implications
  • policy

As the MYM has evolved, however, a number of key themes that cut across these topics have appeared. This article introduces four of them with the hope of stimulating discussion at the meeting. They are the importance of

  • commitment to ongoing network evaluation;
  • frameworks for developing evaluation measures;
  • user-based evaluation; and
  • the impact of information policy.

Commitment to Ongoing Network Evaluation

A typical comment heard from many providers of networked information resources and services is that "there is no time" to engage in ongoing assessments regardless of the type. Thus, to some degree, networked information resources and services evaluation is a method in search of a practice. There appears to be much more talk about the use and importance of evaluating networked information services and resources than actual regular ongoing practice. Often excuses for not doing evaluations seem better developed than ongoing strategies to implement them.

Ongoing evaluation requires time, resources and knowledgeable staff, but perhaps most importantly, it requires a commitment usually an administrative commitment to better understand and improve the services and resources provided via the network. Among other things, commitment must include a previously agreed upon budget that supports ongoing (not a one-shot-only effort) evaluation and assessment processes. A general rule of thumb is that at least five to seven percent of a project budget should be dedicated to ongoing (formative) and final or summative evaluation and assessment. How can such commitment to improvement be instilled in providers and developers of networked information resources and services? As we speak, a dedicated line in the project budget for evaluation occurs rarely at best and not at all at worst.

And there are still many questions that evaluators must address. What models for evaluation efforts can be proposed that may offer the best "return on investment" for the resources committed to such evaluations? What types of evaluation can best improve specific aspects of networked information resources and services? What arguments as to the benefit(s) of evaluating networked information services and resources will move administrators/decision makers to conduct regular and on-going evaluation efforts?

Frameworks for Developing Measures

Measures that describe networked information services and resources can be complex. Based on a number of completed studies, the authors, for instance, have suggested one approach. The figure below shows our framework. Space does not allow detailed discussion, but basically the framework suggests that there are a number of network components, such as technology or management support, that can be assessed in terms of a number of different criteria, including efficiency, effectiveness and usefulness. Two of the authors of this article, Bertot and McClure, provide a more detailed discussion of this framework in "Measuring Electronic Services in public Libraries: Issues and Recommendations" (Public Libraries, 37[3[3]: 176-180).

While this measurement approach may not be appropriate for all types of assessments of networked information services and resources, it is a beginning point for analyzing which components of the network can be assessed in terms of which criteria. Moreover, these approaches may take either the formative or summative strategies mentioned above. At the meeting we hope to explore other frameworks, models or conceptual approaches evaluators can develop for assessing networked information resources and services. How can such models capture the complex and dynamic nature of these networks?

User-Based Evaluation

There also appears to be an evolving discussion on the role of user-based evaluations of networked information resources and services as opposed to system-based, technical evaluations. A number of the papers and panels for the meeting address this issue head on. A user perspective should consider the culture of the communities and sub-communities involved, the relationship between community norms and the use of networks, and the individual needs and uses of the network by the users.

A user perspective assumes that technical criteria alone should not drive the design and implementation of information technologies (IT). Rather, IT design and implementation should take into account the particular communication behavior, information use patterns and work environments of both users and potential users. Since networks, ultimately, are designed to meet user information needs, it seems obvious that evaluations should contain a user-based component.

User-based approaches for evaluating networked information services and resources are critical if we are to determine the degree to which those services meet user needs and actually assist users to be more productive in a variety of contexts. They are essential for successful planning and development of networks and the provision of networked information resources and services. How can we better integrate technical and user-based evaluations? Which types of user-based evaluations can best inform which types of technical assessments and vice versa? Perhaps most importantly, how can network designers learn from users and better evolve services to meet user needs?

Impact of Information Policy

National information policies have an ongoing and pervasive impact on the development of and access to networked information services and resources. Increasingly, a range of information policy topics, such as privacy, intellectual property rights, security, records management, access, transborder data flows, copyright and tariffs, have direct impacts on the types of technologies and services that Internet Service Providers (ISPs) can provide. Issues related to competition in different communications markets as a result of the Telecommunications Act of 1996 (PL 104-104) also have direct impacts on provision of networked services and resources.

Oftentimes, assessments of networked information resources and services fail to consider the national (or state) information policy system and how that context affects the delivery of networked services and resources. Evaluators may not be aware of some of these policy guidelines, nor might they be aware of the degree to which these guidelines have a direct effect on services provided by the ISP. Developing evaluation methods that integrate a range of perspectives, such as a policy perspective, for example, appear necessary to obtain "the big picture" in assessing networked services and resources. To what degree do network assessments take into consideration this broader information policy context? How can assessments of networked information resources and services better consider such information policies?

Expanding the Knowledge and the Practice

Clearly there are other key themes than the four identified here to discuss at the meeting. But these themes do suggest the importance of developing and implementing evaluation techniques for networked information resources and services that rely on a multi-dimensional perspective. Due to the complex technological nature of networks, user behaviors associated with network use and access, and a range of organizational and external factors (such as information and telecommunications policy) that are constantly changing, such an approach is essential. Perhaps due to this complexity, the reality is that hard evidence of impact or even of methodologies that can demonstrate the "success" of providing information resources and services in a networked environment are difficult to find.

If the evaluation of networked information resources and services is to be improved, there is a need to expand both the knowledge underlying evaluations and the practice of conducting them. Evaluation researchers must leave the confines of their offices and the trappings of academia; practitioners must recognize the need for and techniques by which to conduct successful assessments of networked information resources and services. Finally, both groups then need the support of administrators to conduct such evaluations and begin an iterative process whereby evaluations improve the overall quality and success of networked information resources and services.

The papers and panels at the ASIS Mid-Year Meeting will continue and extend both our knowledge and our practice of understanding networked information resources and services. The extended abstracts for the meeting are available at http://www.asis.org/Conferences/MY99/
and provide an outstanding context for extending this discussion. The papers from the meeting, which will be edited by the authors for fall publication, will also extend this discussion. We look forward to participating with you in this exciting event!



John Bertot
University at Albany

Charles R. McClure
Syracuse University

Carol A. Hert
Syracuse University

Committee Members

Anne M. Buck
California Institute of Technology

Michael Crandall
Boeing Technical Libraries

Geoffrey Ford
University of Bristol

Edward Fox
Virginia Tech

Paul Kantor
Rutgers University

Jeffrey Pomerantz

Sarah Allen
Logistical support

Charles R. McClure is Distinguished Professor of Information Studies at Syracuse University School of Information Studies and can be reached at cmcclure@mailbox.syr.edu.

John Carlo Bertot is associate professor at the School of Information Science and Policy at SUNY Albany; he can be reached at jcbertot@cnsunix.albany.edu.

Carol A. Hert is assistant professor at the School of Information Studies at Syracuse University; her e-mail address is cahert@mailbox.syr.edu. They are program co-chairs for the ASIS 1999 Mid-Year Meeting



How to Order

@ 1999, American Society for Information Science