By James Turner, professeur adjoint
École de bibliothéconomie et des sciences de l'information
Université de Montréal
Montréal, Québec
Previous research on shot-level indexing of moving-image documents identified the terms supplied most often by participants to describe a selection of shots from the National Film Board of Canada’s stockshot collection. The most popular terms supplied by participants in the study were compared with the terms assigned by professional indexers for these shots in the source files. Records for some of the shots used in the original study came from the stockshot library’s computer database, and the remaining records came from its older card file. Since the level of indexing is specific in the database and more general in the card file, direct comparison is not possible. However, in both files a high degree of correspondence was found between the most popular terms named by participants in the study and the occurrence of these terms either in the indexing or in the written description of the shots found in the source files. This is encouraging in the context of making collections of moving images available online because it indicates agreement between the terms users think of when searching film and video shots and those indexers assign to them. This suggests that indexing collections of art images at the pre-iconographic (ofness) level in addition to the iconographic (aboutness) level would help improve retrieval rates.
Research to date in the realm of indexing image documents for purposes of storage and retrieval has been concentrated in the area of collections of art. Probably this is due to the importance of these collections as a cultural resource, and to the long-recognized value of art collections. However, non-art images are also collected, and they are associated with a different set of indexing problems. Within this second area are moving image documents which need to be indexed at the shot level, such as are found in television archives and stockshot collections for both film and video.
A study was conducted to gather evidence which would suggest appropriate theoretical directions for providing subject access to still and moving non-art images in storage and retrieval systems [1]. Participants screened a videocassette containing stock footage and for each shot supplied words or phrases which they felt were obvious keys for retrieval at a later date by themselves or others. Analysis was conducted on the most popular terms supplied for each of the shots on the tape. One of the significant findings was that across all the 181 participants and all 44 shots, on average almost 60% of the participants named the most popular term for each shot. Another important finding of the study was that participants overwhelmingly supplied pre- iconographical (ofness) terms rather than iconographical (aboutness) terms, for the most part simply naming the objects and events they saw in the shots.
The present paper is concerned with a follow-up study in which the most popular terms supplied by participants in the study were compared with the terms assigned by professional indexers in the source stockshot collection in order to discover the degree of correspondence. If there is agreement, then indexers are making access points available which correspond to user needs.
The stockshot collection at the National Film Board of Canada contains over 150 000 shots, for the most part stored in the original negative form. Information concerning 18 000 of these shots is stored in an online database, in use since 1988, and information about the remaining 132 000 shots is stored in a card file, started in 1941 and closed in 1988. The sample taken for the original study was representative of the entire collection. Records for 11 of the shots came from the computer database, and the remaining 33 records came from the older card file.
Since the level of indexing is specific in the database (naming significant people, objects and events seen in the image) and more general in the card file (giving the equivalent of subject headings in library catalogues), direct comparison of the indexing is not possible. Interestingly, however, the significant people, objects and events recorded in the card file indexing are often named as subject headings, reinforcing the idea that in practice it is difficult to separate the pre- iconographical and iconographical levels of description in subject indexing [2].
The first part of the present study is concerned with whether the most popular term for any given shot is included as in indexing term or part of an indexing term either in the stockshot database or the card file of the source collection. The second part is concerned with whether the most popular term is also included in the written running description of the content of each shot which forms part of the catalogue record in both the online database and the card file.
The most popular terms for all the shots were compiled and this list was used as the primary reference for comparison with information found in the database and in the card file. Identifying the most popular term for each shot is not straightforward because for most shots, more than one term achieved the highest score. Of the 44 shots used in the study, 17 had a single most popular term; 16 shots had two terms competing for the distinction; 11 shots had 3 or more terms competing. The greatest number of terms competing for the distinction was 9 (one shot). Any of the terms which achieved the highest score for each shot was used for comparison against the data.
In the original study, the terms were analyzed using the first significant word of each descriptor supplied, in truncated form, following a method used by Furnas, Landauer et al. [3] and based on the idea that in a computer retrieval system the search would probably be conducted in this way. However, the technique of collapsing terms to the first significant word was not strictly adhered to in order to keep concepts separate when verbal expressions are identical, and to deal with variant spellings, illegible handwriting, and editorial comments found in the data. To accommodate these problems, a number of arbitrary rules were devised at the data entry stage.
In order to compare the most popular term against the indexing record for each shot, three files were created. For each of the 11 shots in the computer database, the indexing terms assigned were listed, including indexing for aspects other than strictly subject indexing, such as time periods, geographic descriptors, and camera angles, which in the database are expressed using classificatory or other structures and represented in lookup tables. For the 33 shots in the card files, the records were photocopied. The recto of these records includes a running description covering the action in entire reel of film, as well as shelf location and other identifying information. The verso includes descriptors assigned to the reel and used to produce card sets for subject access. Finally, the list of most popular terms identified for each shot was divided into two sections, one for the shots recorded in the database and one for the shots with card records.
In the first part of the study, a match was sought between any of the most popular terms for each shot and any of the indexing terms assigned for the shot. In the second part of the study, a match was sought between any of the most popular terms for each shot and any word in the running description for the shot. The ratio of successful matches was then calculated.
A comparison of the most popular terms with the indexing terms assigned by professionals and the occurence of the terms in the running description is presented in Table 1. The most popular terms for each shot used in the original study are given in the column labeled "Most popular term". In cases where more than one term achieved the highest score, all the competing terms are given. They are presented here as stemmed for the purposes of the original study, while the terms given in the columns labeled "Indexing term assigned" and "Appeared in description as" are given as they appear in the source collection records.
It is noteworthy that generally the shots converted from the card file to the computer database are indexed from the written description, not from the image. On the other hand, in the study the participants saw the image but not the written description. The indexing terms shown are not exhaustive. In some cases several other terms are assigned, but are not necessarily relevant to the 10-second segment of the shot seen by participants in the study, or are not given here because they are not related in any way to the most popular term from the study. The absence of any word representing a concept related to the most popular term supplied by participants in the original study is shown in the table as an em-dash (-).
As regards the analysis related to indexing terms, how successful users would be be in retrieving the shots in this sample is of course a function of the match between their inputs and the indexing terms assigned. For the shots in the database, a match would occur in 9 cases out of 11, or 82% of the time. For the shots in the card file, a match would occur in 28 cases out of 31 (two shots were eliminated from this analysis because the indexing for them was not available), or 90% of the time.
As regards the analysis related to the running descriptions of the action in the shots, how successful users would be is a function of the match between their inputs and the presence of that term in the running description. For the shots in the database, a match would occur in 9 cases out of 11, or 82% of the time. Interestingly, the two shots for which a match does not occur are the same two as with the comparison with indexing terms. For the shots in the card file, a match would occur in 25 cases out of 33, or 76% of the time. A summary of these results is given in Table 2.
The success rate is encouraging and shows that at least as far as concerns the test database, there is a high rate of indexer-requester consistency [4]. However, two factors which foster the high number of matches need to be explained.
The shots used in the original research were chosen for the levels of agreement they engendered among indexers in a categorization exercise of 200 shots compiled from two separate random samples from the database and the card file in the source collection. The indexers were given the shot descriptions and a list of 13 categories (12 subject categories and "Other") and asked to place each shot in a single category. The 51 shots which all 11 participants in the exercise placed in the same categories was used as the pool from which the 44 shots which appeared on the research tapes were taken. Thus there is some likelihood that the high degree of successful matches found in the present study has to do with a kind of built-in subject unity inherent in the choice of shots.
Secondly, since the data (consisting of terms given to describe stock footage) showed a strong bias in favour of pre-iconographical or ofness terms, more success in matches with the indexing in the source database can be expected than if the research had been conducted with art images. Databases of art images are almost invariably indexed only at the iconographical or aboutness level of description. Shatford Layne [5] wonders whether there might be more agreement on the principal and more objective aspects of the subject of an image, and less on the secondary and "subjective" aspects. The high level of agreement between user and indexer in the results obtained here suggest that this is the case. Milstead’s observation that "the known high level of inconsistency in indexing must arise at least partly from differences in perception... of the ‘aboutness’ of the document" [6] does not apply to databases of moving image documents indexed at the shot level, because individual shots are not about anything until they are used in some production which gives them context. To complicate matters, often it is impossible to distinguish between ofness and aboutness for purposes of indexing [7].
The results obtained in this study provoke the gnawing question: Since there is a high rate of matches between user inputs and their occurrence in the running descriptions, do we need indexers? The data suggest that indexers name the obvious, as did participants in the study. However, the match represents only a single avenue into the data. Indexers provide access points into less obvious aspects of a shot which may be of interest to another user. In addition, they provide tools which help the user sort out the vocabulary by showing semantic relationships between terms, suggesting related material which might be of use. It has been amply demonstrated that much of the information useful for retrieving an image is not available from the image itself. In a database that takes many facets into account, the important information for some requests may not be available from the running description either. As I have shown elsewhere [8], for many searches it can be more efficient to avoid the subject access points altogether.
The difficulties of subject indexing in general are numerous and complicated, and it is doubtful whether they can ever be resolved satisfactorily. However, our question about the need for indexers becomes more pertinent when targeted more precisely to the subject indexing of non-art images. The results of the present study strongly suggest that automating subject access to non-art images indexed at the shot level can be successfully achieved. While the level of matches achieved is high in the present results, no special conditions favoring this were present in the creation of the source data. If in future those who select the material and write the running descriptions are made aware that they are also indexing the shot for later retrieval, they can arrange to include in the descriptions any terms thought useful for retrieval. Along with other techniques such as browsing [9] and "enriched access tools" [10], indexing derived from running descriptions may prove to be an important member of a team of useful retrieval strategies available to the user.
The results of the present study strongly suggest that the question of using running descriptions for moving images to derive subject indexing for those images should be further investigated. Moving image databases for stockshot and archival purposes need to include descriptions of the action in each shot, and these shot-level descriptions are invaluable to the user of a moving-image database since they are an efficient way to reduce retrieval sets to a manageable shot list for viewing [11]. The results of the study apply to non-art moving images but not to "ordinary" (i.e. non-art) still images, since these latter are usually not accompanied by running descriptions.
Nor do the results apply to collections of art images, still or moving, but they suggest that indexing collections of art images at the pre-iconographic (ofness) level in addition to the iconographic (aboutness) level would help improve retrieval rates, at least for some users. Successfully retrieving art images from information systems is probably more dependent on access points such as the name of an author or a title, but being able to name persons, objects, and events that occur in art pictures would make possible search questions that cannot be asked of a system without this kind of indexing. As is the case in other areas of study, it has become evident that rather than discussing which strategy is better than the others, we should be investigating which combination of strategies yields the best results for various purposes.
Acknowledgements: This work was completed with the financial assistance of the Comité d’attribution des fonds internes de recherche (CAFIR) of the Université de Montréal, and with the help of Johanne Lacroix, who served as research assistant for the study.
1 No longer considered politically correct, the derogatory term Eskimo is replaced by the term Inuit in the online database. In the present case, the term Eskimo appears in the indexing but not in the part of the description related to the shot. When the record for this footage is included in the online database Eskimo will be changed for Inuit in the indexing. In that sense a match with the most popular term Inuit occurred here.
2 In 1988 a new policy was adopted in the stockshot library. The card file, begun in the early 1940s, recorded information in English only; after 1988 descriptions for shots from French-language productions were in French, and those from English-language productions in English. The indexing remained in English for all shots. In 1991 the policy changed again, and now all the database records have both descriptions and indexing in both languages. The card records for these shots being caught in the interim policy, the actual term in the description is in French while the indexing descriptor is in English.
3 The card in the file is a copy and the indexing, normally on the verso, does not appear.
4 The card for this shot has been misplaced since the description was copied for the study. Thus the appearance of the term in the description is known from other records of it, but the indexing could not be found.
(1) James Ian Marc Turner, Determining the subject content of still and moving image documents for storage and retrieval: an experimental investigation (PhD thesis, University of Toronto, 1994).
(2) Ibid., 151-153.
(3) G.W. Furnas, T.K. Landauer, L.M. Gomez, and S.T. Dumais, "Statistical semantics: analysis of the potential performance of key-word information systems", The Bell System Technical Journal, 62, no. 6 (July-August 1983) 1753-1806.
(4) Raya Fidel, citing W.S. Cooper, " Is interindexer consistency a hobgoblin?", American Documentation, 20 (1969) 268-278, in "User-centered indexing", Journal of the American Society for Information Science, 45, no. 8 (1994) 572-76.
(5) Sara Shatford Layne, "Some issues in the indexing of images", Journal of the American Society for Information Science, 45, no. 8 (1994) 583-8.
(6) Jessica L. Milstead, "Needs for research in indexing", Journal of the American Society for Information Science, 45, no. 8 (1994) 577-82.
(7) Turner, Determining the subject content..., 14-16.
(8) James M. Turner, "Indexing non-art pictures: some issues", Proceedings of the American Society of Indexers / Indexing and Abstracting Society of Canada conference, Montreal, June 8-10, 1995 [in press].
(9) Brian O'Connor, "Browsing: a famework for seeking functional information", Knowledge, 15, no. 2 (1993) 211-31.
(10) Brian O'Connor, "Concepts and techniques for use-specific representations of documents", Libres: Library and Information Science Research Electronic Journal, 4, no. 4 (December 31, 1994).
(11) James Turner, "Representing and accessing information in the stockshot database at the National Film Board of Canada", Canadian Journal of Information Science 15, no. 4 (December 1990) 1-22.
Comparison of the most popular terms with indexing and descriptions
______________________________________________________________________
Shot Most popular Indexing Appears in
number Term term assigned description as
1. Computer database
04 radio radio radio
06 construct, prefab construction prefabricated
07 chicken chickens chickens
11 prairie, field landscapes -
17 winter, snow winter winter
[time period descriptor]
18 lake,aerial,geography aerial aerial
[camera angle descriptor]
19 lake, island landscapes, rivers -
21 stadium stadiums stadium
24 ship,ferry,ocean,boat ferries ferry
39 cat cats cat
42 airplane,plane,seaplane, seaplanes seaplane
transport
2. Card file
01 flag ceremonies¯NATO forces flag
02 child, family, Inuit Eskimo 1 -
03 bird birds birds 2
05 bird birds -
08 fish, fisherman fishing, fishermen fish,fisherman 2
09 mountain, rock mountains mountain
10 ship,box,man,cargo,open,work ships ship,men,work
12 shop,street,window, streets,store shop,shoppers,
store, people street, windows
13 train,forest,mountain train, mountains train, mountains
14 grocery, cash cash cash
15 bird birds -
16 train, freight trains, freight train, freight
20 winter, snow winter winter
22 buffalo buffaloes buffaloes
23 work,man,fish,fisherman,net fishing fishermen, nets
25 rock, mountain mountain -
26 woodpecker, bird birds woodpecker
27 old, log log log
28 building, urban buildings buildings
29 bird birds -
30 university universities university
31 duck, bird ducks, birds -
32 fireman, firefighter firemen firemen
33 moose,forest [indexing not available]3 moose,forest
34 mountain [indexing not available]4 mountain
35 sun sunrises sunrise
36 ski ski skiers
37 factory, bottle dairies, food -
38 computer computers computer
40 deer deer deer
41 native,banquet,Indian,meeting banquets,Indian -
senior,people,gift,old,elderly
43 cow, cattle, farm farms cattle
44 queen royalty queen
____________________________________________________________________
TABLE 2
Summary of results
_____________________________________________________________________
Shots in % Shots in %
the database the card file %
Term appears in indexing 9/11 82 28/31 90
Term appears in description 9/11 82 25/33 76
______________________________________________________________________
James Turner