B U L L E T I N
Data Collection for Controlled Vocabulary Interoperability—Dublin Core Audience Element
Joseph Tennis is a doctoral student in the Information School of the University of Washington. He can be reached by e-mail at firstname.lastname@example.org
This paper outlines the assumptions, process and results of a pilot study of issues of interoperability among a set of seven existing controlled vocabulary schemes that make statements about the audience of an educational resource. The notion of audience for the study was defined in terms of the semantics of the Dublin Core metadata element of the same name: "A category of user for whom the resource is intended." The study used a data collection technique, card sorting, to see how non-expert users (1) sorted terms in the seven vocabularies into relationships and (2) what their thought processes were in sorting these terms.
The need for controlled vocabulary interoperability is a pressing concern for the education community as well as many others. In particular, the current study was informed by the need of the Dublin Core Education Working Group (www.dublincore.org/groups/education/) to explore the possibility of a high-level switching language in an application profile for the Dublin Core Metadata Initiative (DCMI) audience element. An abundance of educational resources exists, many of which are available in the networked environment. Yet, there are various conceptualizations of the domain in the form of different controlled vocabularies that limit access. Controlled vocabulary interoperability would allow these different conceptualizations to remain intact, thereby serving local needs while allowing users to navigate across collections and exploiting the intellectual network of resources available.
Assumptions of Controlled Vocabulary Interoperability
Controlled vocabulary interoperability is a problem that is complex and has a long history in information science. From the 1960s onward there are discussions in the literature of attempts to reconcile at least two controlled vocabularies from the same domain. The result of this body of research has been the creation of mapping and switching techniques. Mapping attempts to match one-for-one the terms of each controlled vocabulary. This technique assumes that you can find the same meaning (in its extension and its intension) between two terms in two different controlled vocabularies. The other result, switching, advocates a third intervening language be constructed that can switch between the two controlled vocabularies. This also assumes that the problems of meaning can be reconciled between three terms: two controlled vocabulary terms and one switching term. Examples of these types of interoperability are given in Tennis (in press. See For Further Reading.)
Lancaster presents the classic argument against the perfect actualization of interoperability. Lancaster states that the structure of subject access systems confounds any seamless switching. The problems inherent in switching between two vocabularies are differences in (1) overlap of subject matter, (2) specificity, (3) degree of pre-coordination and (4) hierarchical, synonymous and other relationship structure. For Lancaster each term is defined in relationship to other terms in the controlled vocabulary. As a consequence the meaning of the term may be more broadly or narrowly defined (1 and 2 above) or the placement of that term below another term may shape its meaning implicitly (3 and 4 above).
Therefore the system that allows interoperability between two (or more) controlled vocabularies must reduce or eliminate structure while maintaining meaning. By reducing or eliminating the structure of terms from controlled vocabularies, it may be possible to shift interoperability toward a pragmatic management of these terms as they exist for the user and the user's needs.
The questions and assumptions in interoperability are philosophical. What is the best approach? What is the best-reasoned design of an interoperable space? Interoperability can become solely an object of investigation and not necessarily an area of development of best practices and principles. Since the end goal of this study is to develop a technique or, perhaps, a set of techniques to support interoperability among existing controlled vocabularies used with education materials, the issues in the study were cast as a problem of information management that had to be informed through empirical analysis. Given this problem, what techniques can be employed to inform a decision about how to manage the interoperability between these education-focused controlled vocabularies? Specifically, what kind of data do we need to begin the iterative design of a term management space that will allow for interoperability? One data collection technique available is card sorting.
Assumptions of Card Sorting
Card sorting, or just sorting, is used by various disciplines to examine how individuals organize a given set of cards. Often the cards have terms on them. Sometimes other materials are sorted. For example, Carlyle (2001) sorted different instances of one work. The results of the sort are interpreted as a user's (sorter's) conceptual arrangement of the given universe of objects. In information science and in usability work, sorting is a tool that can be used to help inform the design of online displays. Carlyle reviews various aspects of sorting. She also points to Fidel's suggestion that data derived from users should be used to inform systems design. Carlyle also suggests that various user groups, not just specialists in a field, should be part of the card sort.
It is with these assumptions that this pilot study began. We wanted to investigate the viability of a card sort technique to inform an aspect of systems design – specifically the design of an interoperability space between various controlled vocabularies used in education. The results would allow me to evaluate the method as a data collection technique for interoperability and to modify the card sort technique if necessary.
The Card Sort
The card-sort pilot study was a first attempt at reconciling seven different controlled vocabularies in the field of education. The terms sorted were audience types. Audience types are those people or groups that might use the resources in an education information store. For example, these people may use lesson plans, simulation tools or instructional texts. Audiences include parents, teachers, vocational educators, counselors, doctors, etc. Many are very general terms in themselves. Yet their meanings are restricted because they are deployed in the context of the education domain. Thus counselors are specific types of counselors – those whose work focuses on student activities and problems.
A purposive sample of terms was chosen from controlled vocabularies developed by Education Network Australia, European Treasury Browser, Gateway to Educational Materials, Instructional Management Systems, Australian Government Locator Service, UK National Curriculum and U.S. Department of Education.
The terms in the purposive sample were considered unique and more general than those left out of the sample. Due to the experimental nature of the card sort a limited set of 37 terms was selected (see box). The participants were asked to sort these cards into relationships that made sense to them. They were told that the terms were from the domain of education and represented kinds of users (audiences) of educational materials. The participants were also asked to label any groups or subgroups that resulted from their sorting. The purpose of this labeling was to explore potential high-level vocabulary terms that might be used to represent the groups or subgroups. While the participants were sorting, they were asked to talk aloud so that their thought processes could be better understood. For this pilot nine participants volunteered their time.
Results of the Card Sort
This pilot card sort generated three types of data: (1) the piles of cards (with high-level vocabulary labels supplied by the participants); (2) a transcript of the talk-aloud; and (3) observations of the act of sorting that may prove quite instructive.
Sort Results: The piles of cards with their labels were entered into Microsoft Excel as occurrence matrices where the occurrence of a term with another term (to form a group) was indicated in the matrix with a "1." A term that did not appear with another term was marked with a "0." The matrices were totaled. A number of cluster analyses were conducted on the total. The cluster analysis describes the aggregate result of the sorting task by all nine participants. However, individually each pile informs us of details that are lost in the aggregate.
The labels provided by the participants should be examined conceptually rather than literally. The participants were encouraged to be casual through the talk-aloud protocol. This resulted in some labels for groups that reflected participant moods or humor. However, these labels still carry weight as grouping terms on the conceptual level. Two participant labels appear below. Keep in mind that the participants sorted all 37 cards into these groups and labeled them.
Participant "A" Card Group Labels: Education Consumers; Teachers; Administrators; Outside
Participant "B" Card Group Labels: School Administration; School Publishing; Teaching Staff; Students, Alumni and Families; Non-Teaching Staff
As can be seen in this data the two participants do not agree on the number of groups or the names of those groups. That is why it is helpful to see the results in the aggregate.
The aggregate picture of the card sort, when examined using a cluster analysis (Ward's Method in the SPSS program) looks like this:
Cluster A: Animateur; Media Specialists; School Aides; Librarians
Cluster B: Educational Administration; Political Decision Makers; School Publisher; Author
Cluster C: Inspector; School Leadership; Managerial Staff; Manager; School Personnel Worker; Curriculum Supervisor; Non-teaching Staff; Technology Coordinator
Cluster D: Guidance Officer; Speech Therapist; School Nurses; Counselors; School Psychologists; School Doctor
Cluster E: Graduates; Learner; Alumni; Stopouts; Dropouts; Students; Families
Cluster F: Teachers; Trainer; College/University Instructors; Educationalists; Tutors; Teacher Interns; Vocational Educators; Adult Educators
This aggregate fits much of the criteria for a good clustering of the terms. These terms fit first and foremost with a common-sense analysis. These terms do go together and they are similar to the individual participant sorts. However, the agglomeration schedule generated by the Wards Linkage process of SPSS shows that the optimum number of clusters is four. There are six clusters in this data. The problem lies with groups A through C. The participants were very creative when sorting these terms – distributing them across many groups. They were not as creative when sorting D, E and F. As a consequence these terms appear most often together and, therefore, fall naturally together in the cluster analysis.
There is additional work that can be done with this data. Cluster analysis is full of variations on method and of controversy over applicability. In this study, cluster analysis was used as a descriptive technique to find structure in the data gathered. It was not in any way intended as a predictive analysis. Because of wide ranges of interpretations of cluster analyses, it is encouraging to see such strong evidence in the common-sense analysis (the coherence of the six groupings given above). It seems that this data could inform the design of an interoperability space.
Results of the Talk-Aloud
The transcript of the talk-aloud is still being analyzed. However, from preliminary examinations of the data, it seems clear that the participants wanted to construct a meaning for a term in relationship to the other terms available. Often the participant was confused, expressed this confusion and made decisions without certainty.
Results of the Observation
Across all participants common phenomena were observed. First, each of the participants laid the entire set of cards out before him or her. From this undivided universe of 37 cards, they began the process of sorting. A second common process in this sorting task was a vacillation between sorting from the bottom up and sorting from the top down. Some categories grew from lumping like things. Others were made from dividing the unlike groups of terms, then further dividing. However, neither approach (top-down nor bottom-up) was used exclusively by any of the participants.
Conclusion – Data collection and Interoperability
As a data collection technique, card sorting provides the researcher with various types of data – some of it very rich. As part of the decision making process for systems design, a card sort should be only one of many sources of information. Collecting data from card-sort techniques will not solve the philosophical problems of controlled vocabulary interoperability. As Miller notes (2000) much work needs to be done to understand what is wanted from interoperability in general. From a menu design perspective, the pragmatic management perspective, card sort techniques allow those in need of solutions a way to inform their decisions, as McDonald and Schaneveldt show (1988).
More research is needed into the particularities of card sort data collection in this venue. What role does the expertise of the sorter play in the resulting piles? Hayhoe (1990) suggests that it is beneficial to have both novices and experts shape the resulting aggregate for cluster analysis. How do terms within the domain of education like Animateur, unfamiliar to some, influence the analysis? In this particular case, the talk-aloud and observation data helped to identify the problems the participants had with Animateur, whereas the aggregate analysis of all of the piles hid this problem.
From a pragmatic point of view, it is necessary to test this data collection technique a number of times and in different contexts. If card sorting is to become a viable method for aiding controlled vocabulary interoperability it should be rigorous yet easy to perform. It should generate enough data to give a coherent picture, but not create paralysis from data overload.
Following a close analysis of this pilot study, we hope to design and carry out a more robust research project. There is much work to be done in controlled vocabulary interoperability if metadata initiatives are to take advantage of the networked environment. Linking intellectually as well as hypertextually to stores of information will require interoperability. In order to ensure interoperability those who work with metadata will want best practices information about how to construct an interoperability space from different controlled vocabularies.
Miller, P. (June 2000). Interoperability. What is it and why should I want it?" Ariadne, 24. Available: www.ariadne.ac.uk /issue24/interoperability/intro.html.
Vocabulary Mapping and Switching Languages
Dahlberg, I. (1996). Compatibility and integration of order systems 1960-1995: An annotated bibliography. In Compatibility and Integration of Order Systems: Research Seminar Proceedings of the TIP/ISKO Meeting, Warsaw, 13-15 September, 1995. Warsaw: Wydawnictwo.
Lancaster, F. W. (1986). Vocabulary control for information retrieval. (2nd ed.) Arlington, VA: Information Resources Press.
Tennis, J. T. (in press). Layers of meaning: Disentangling subject access interoperability. In Efthimiadis, E. (Ed.) Advances in Classification Research, 12.
Sorting and User-Centered Design
Carlyle, A. (2001). Developing organized information displays for voluminous works: a study of user clustering behavior. Information Processing and Management, 37, 677-699.
Fidel, R. (1994). User-centered indexing. Journal of the American Society for Information Science, 45, 572-576.
Hayhoe, D. (1990). Sorting-based menu categories. International Journal of Man-Machine Studies, 33, 677-705.
McDonald, J. E. & Schaneveldt, R. W. (1988). The application of user knowledge to interface design. In Guindon, R. (Ed) Cognitive science and its application to human-computer interaction (pp. 89-338). Hillsdale, NJ: Lawrence Erlbaum, pp. 89-338.
Copyright © 2003, American Society for Information Science and Technology