The QBIC Project in the Department of Art and Art History at UC Davis

Bonnie Holt, Ken Weiss

University of California, Davis, California

Wayne Niblack, Myron Flickner, and Dragutin Petkovic

IBM Almaden Research Center, IBM Corporation, San Jose, California


Abstract

The Art and Art History Department at the University of California at Davis is testing QBIC™ (Query By Image Content, image query software from IBM) as a tool for managing and retrieving images from on-line collections of digitized artwork. We have created two QBIC image databases. One is a website linked to the department's home page that allows students and the public access to images of artists who are teaching in the department at Davis, and the other is a non-web version that allows students access to a database of art images from an X-terminal in the Art Department Library. Art images are often difficult to describe precisely in words, a limitation addressed by QBIC's ability to perform searches based on how an image looks. The user can perform queries based on example images. A thumbnail image is displayed, and the system can search for other images with similar color, texture or overall layout. The user can also use graphical tools to specify arbitrary characteristics such as a color histogram: 20% of a specific shade of blue, 30% of a shade of green. The search will return results in the form of thumbnail images arranged in descending order of match to the user's query. Text attributes such as the artist's name or media can also be used to restrict the search. The web version of QBIC continues research funded by IBM that began in 1993 when the department began testing QBIC on a pilot database of art images from the department's heavily used slide library.



The QBIC Project

The original motivation for the Art Department's involvement in this project stemmed from the way the slide collection is used. The Slide and the Art Libraries in the Department of Art and Art History at UC Davis house a collection of more than 200,000 slides and approximately 90,000 mounted art reproductions. These collections serve as a unique visual resource for faculty research and teaching within the department and across the curriculum. Our collections are organized according to medium, alphabetically by country, chronologically by date, and then alphabetically by artist. The slide librarian is in the early stages of creating an electronic catalog for the slide collection, but its accessibility to patrons is even now years away. Access to both collections is based on the labels on the file drawers and the labeled file cards in the drawers. In the slide collection, especially, there are works of art that strain logic. Where do you file slides of conceptual artist Victor Burgin's work? Should it be under "New Media" or should it be filed under "photography?" Especially in contemporary art the terms that are commonly used are in a state of flux. QBIC resolves this by allowing images to be classified visually and without "naming," and images can be found by users of the database without the users agreeing on terminology.


Librarians from both libraries frequently deal with requests from patrons based on the content of the image rather than a particular artist's work. The search may be for a particular brushwork, an image of an animal, or a particular iconographic element. Typical examples are a studio professor who has assigned a project in a painting class that requires that the students restrict their palette to a pair of complementary colors, or a professor from Wildlife and Fisheries Biology who is researching 'fish in art' for a publication. The Fine Arts and Slide Librarians are both art historians, and are familiar with the collections. Current cataloging would not help to identify either request. In most cases the staff may be able to help the patron find some suitable examples of what they are looking for, but it is a laborious, and sometimes daunting process. It means that the librarians stop what ever they are doing to consult with the patron, and then spend time pulling slides, or reproductions from file drawers, helping the patron refine their search, until they have suitable examples to use. If patrons were able to perform searches electronically, they could do so without depending on the staff. Suppose they could define their search visually, rather than try to describe it, and then browse images without having to pull them from the file. Their search would be more efficient and comprehensive, allowing them to access related images in different media, across geography and time. In addition, this would be more cost effective, would require less staff time, and would reduce wear and tear on the collection.


X-Windows QBIC

We began the project fall quarter, 1992 with an initial database of images that was selected by the art and slide librarians based on their reference log, including "visual problem sets" that addressed repeated requests for images. For example, we started with the request for "nude reclining figures." We were able to identify examples from pre-history to Postmodern, across geography and media, and in differing degrees of abstraction. Examples of other "problem sets" included "text as a texture in a work of art," fish, horses, insects, phallic symbols, and monsters. Since a number of our faculty are interested in issues of "race, class, and gender", we kept these issues in mind in when making our selections. Other additions to the initial pilot study included a selection of archival images of works of art in the Nelson Gallery and Fine Arts Collections at UC Davis, and slides documenting a range of the current work produced by the artists who teach in the department.


We started scanning slides in January, 1993. Because of the limited space that was allocated to us on the RS-6000 that we shared, and the large file size of images, we scanned all images at 1,000 dpi and at 8 bits of color. This allowed us space for only 700 images. Each slide was scanned on a Kodak RFS-2035 Professional Slide Scanner at 1,000 dpi, then converted to 8 bit indexed color to save space. This was chosen as a compromise. Since this early version of the software only supported TIFF and GIF image formats we had to choose between full color at a low pixel resolution, or 8-bit color at higher pixel densities. Since most of our displays were (and still are) capable of only 8-bit color, we decided to use the GIF format to maintain good image resolution while achieving maximum storage capacity. A 4-gig external hard drive was added in 1994 which allowed the database to grow to approximately 1,000 images, and the following year we obtained our own server. Subsequent versions of the software have supported the use of JPEG compression allowing us to store larger files of images scanned at 24 bits of color and 1000 dpi. Our database now contains approximately 2,000 images. The images are color corrected, cropped, and spot toned in Adobe Photoshop 2.5 on a Mac Quadra 950 and saved as JPEG files ranging from 1 to 3 megabytes. The images are then downloaded as binary files to an IBM RS-6000 workstation which runs QBIC under AIX and X-Windows. Each image is associated with a text file that includes the slide accession number, and supporting text, including: artist, title, medium, and dimensions. The database is accessible from an x-terminal in the Art Department Library.


QBIC will search images based on their visual qualities, such as shape, texture, and color. The X-Windows version of QBIC that we are testing on our pilot database of art images in the library allows library personnel to manually outline objects or areas in an images using drawing tools. These outlined areas are stored as "masks" in a file associated with the image, and lets us create detailed "visual indices" of qualities such as paint texture, pattern, and shape, etc. The outlining can be labor intensive, depending on the number of components cataloged, and is admittedly subjective. Classification was done by a graduate research assistant working under the supervision of the librarians. All objects that support our problem test sets were carefully outlined, and interesting shapes within those images were selectively outlined. In addition, the research assistant sampled dominant colors, patterns, and textures from each image. The number of components classified for each image depended on the complexity of the image, and ranged in number from four to ten. During the classification procedure, a visual thesaurus of textures was compiled that can be used as a basis for searches.


The X-Windows version of QBIC is one of several generations of the software that allow whole scene searches based on the general characteristics of the image, or "object" searches, that allow searches for characteristics of shapes or segments that have specific color or texture characteristics, and even allows searches for objects in a specified location in a specified horizontal or vertical format. For example, a search can be created for an oval placed in a vertical format and the software will return a range of "hits" that include high numbers of portraits.


Our results have been quite variable. For example, searches for shapes in fine art images are problematic. Shadows and light, and differences between drawing, painting, sculpture, photography, prints and textiles can all make too much of a difference in the same subject for the software to pick up similar conceptual subjects that have slightly differing shapes. Shapes in fine art overlap and have ambiguous contours. These differences make the shape search less accurate. A good example is a query for images of horses. The database contains approximately thirty-eight images in which horses have been classified. These include various media: sculpture, painting, and photographs. The horses are positioned from the side, front, back, three-quarter view, some running, some at rest, etc. Because of the difference in views, the software cannot discern between horses if these views are significantly different. While the human eye can know that it is a horse, electronically, the shape is not the same as the horse that is queried against. When a query is based on any one of these, generally the software will bring back three or four horses, but also dogs, elephants, and other images with similar appendages. But as horses are found they can be moved to the holding window and each queried upon by example and the search expanded. In a relatively short time, relying only on image content rather than text, it is possible to retrieve as many as thirteen or fourteen horses within five or so queries.


One of our goals was to consider whether the software would be able to successfully perform queries that deal with issues of class, race, and gender. This was a primary reason for carefully classifying all of the faces in the database images. Only the face was outlined, not hairdos or hats. A face search can begin by initiating a shape search and using the drawing tool to create an oval. The software will immediately return a range of images from the database, most of which contain human faces, but also many other related shapes, such as an egg, a plate, or a coin. Images from the search that contain faces of people of color can be moved to the holding area and used as a basis for combined shape/histogram searches. The images held may vary in media, and may cross time periods, and geographical areas. They may range from African ceremonial masks to works by Faith Ringgold, the African American textile artist and painter, to a painting by Winslow Homer. The thumbnail size is not large enough to see some small elements and it is not always clear why an image has been returned as a hit. The software allows the user to easily move back and forth between thumbnails of the whole image and thumbnails of details of the image that contain the classified elements. We have been surprised about content in familiar paintings that is frequently overlooked. Within painting, skin color varies between realistic color and expressive or conceptual color. A photographic image in black and white of a black person is generally different too, and sculpture may be the color of the material used. In spite of these limitations, we have been impressed with our success in retrieving a wide range of images of people of color using the software's ability to browse large numbers of images quite rapidly, by selecting from the thumbnails, and moving those selected to the holding area as a basis for further searches.


After several months of testing QBIC against our relatively small (2,000 images) database we reached some interesting conclusions. QBIC can not replace conventional database tools for thematic searches. It can, however, provide a very valuable adjunct to those tools. In a large database of images we anticipate using some simple keywords to reduce the entire collection down to a manageable subset of potentially interesting images, and then using QBIC to arrange that subset so that the most interesting ones are at the top of the list. In many cases, QBIC's greatest value is as a sorting tool, not a searching tool.


The Web Version of QBIC

This fall we began testing a subset of our database using a Web version of QBIC. (http://libra.ucdavis.edu, and at IBM, http://wwwqbic.almaden.ibm.com/) The new web version of the software is easy to use and has improvements over previous versions, but does not currently include the manual outlining capabilities to identify objects. The software now automatically performs some of these functions by analyzing the digitized image, and adds a new color layout function that allows searches for color in a specific location. Another plus is that it greatly improves access which is now possible for anybody who has a simple WWW browser on any desktop platform.


The Web version of QBIC allows a variety of searches. The user can do a keyword search for an artist's name, media, title word, etc., and the software will retrieve thumbnail images by the specified artist or medium. The user can display an image full size by clicking on the space beneath the thumbnail and the software will display the full image with associated text and the slide library accession number. A similarity search can be done by leaving the keyword blank, then specifying either color or texture, click on a thumbnail of an image. The software will bring back images with similar qualities in descending order of match. The user can combine a keyword search with color or texture, or the user can do a customized search, setting percentages of specific colors and even their placement in the image. For example, a user can can begin a search by typing the artist's name "Hollowell" in the keyword window and the software will bring back the works by painter David Hollowell from the database. [Figure 1] Hollowell's paintings have a distinctive texture because of his use of a "pointillist" technique. To find works with similar textures, a "texture similarity" search can be performed by leaving the keyword blank and setting the search characteristic to texture. Then the user can click on the thumbnail of a Hollowell painting that is displayed and the software will bring back images with similar textures by all artists in the database, arranged in descending order of match following the sample image. [Figure 2]


Figure 1. Results of Text Search on Artist's Name "Hollowell"


Figure 2. Results of Similarity Search Based on a Hollowell Painting

For an exercise in the use of complimentary colors in a painting class, images in the database can be retrieved by clicking on " customized search." The "Color picker parameters" window will appear, allowing the user to designate specific colors and their percentages for the search. The color search grid is based on the Munsell color system and is very precise. The user can choose specific shades of red and green, for example, and their percentages in a composition, and the software will return "hits" in descending order of match. The same function can be used to find images by an artist whose name you do not know, or can't remember, but whose color palate you can visualize.[Figure 3]


Figure 3. Color Picker Parameters Window

The "color layout" feature allows the user to select shades of one or more colors and provides drawing and fill tools to specify the placement and quantity for the search. The user can set the software to look for specific colors in specific areas of the composition and the software will bring back a range of images with those characteristics. [Figure 4] QBIC works much like "spelling check" in a word processing database, presenting a range of images, "do you mean this, or this, or this?" QBIC allows the user to browse great numbers of images in rapid succession, refining their search based on what appears on the screen. It is very appealing to users who are visually oriented, or who find it difficult to describe the exact visual qualities that they are seeking, but they know it when they see it.


Figure 4. Color Layout Search

We would welcome an "object" or "shape" search feature in the Web version of QBIC, however the elimination of the arduous task of hand classifying each of the images more than compensates for that limitation. The Web version also solves the problem that we have had with accessing our database from personal computers because users need terminal emulation software to access the X-windows version of QBIC, and even then, the image display is less than desirable. We would like to expand the size of our database to include all of the slide library holdings. This would be an invaluable resource for faculty and students doing research, and would provide a visual catalog of the library's collection, and ultimately could serve for instructional purposes for seminars and lecture classes on campus. The University is in the process of a major upgrade to the backbone network, with the Art Building scheduled for conversion this summer. When this system is complete the technology will be in place to set up an "intranet" on the Davis campus, with dedicated network connections in classrooms that could have password security to restrict access to our database. We are exploring funding sources to proceed in this direction.


REFERENCES

Barber, R., Equitz, W., Lee, D. and Niblack, W. (1994) User Documentation for QBIC. IBM Almaden Research Center.

Besser, H. (1992). " Adding an Image Database to an Existing Library and Computer Environment: Design and Technical Considerations." In Studies in Multimedia: State-of-the-Art Solutions in Multimedia and Hypertext, Proceedings of the 1991 Mid-Year Meeting of the Association for Information Science, San Jose, California. (pp. 31-45). Learned Information, Inc., Medford, NJ.

Besser, H. (1990). Visual Access to Visual Images: the UC Berkeley Image Database Project. Library Trends, 38, 787-798.

Cawkell, A.E., (1993). Picture-queries and picture databases. Journal of Information Science, 19, 409-423.

Cawkell, A.E. (1992). Imaging systems and picture collection management: a review. Information Services & Use, 12, 301-325.

Holt, B. (1994). Retrieving art images by image content: the UC Davis QBIC project. Aslib Proceedings, 46, October 1994.

Holt, B., Hartwick, L., and Vetter, S. (1995). Query by Image Content, the QBIC Project's Applications at Davis's Art and Art History Departments. Visual Resources Association Journal, 22, no. 2, Summer, 1995.

Niblack, W. and Flickner, M. (1993). Find me the pictures that look like this: IBM's Image Query Project. Advanced Imaging, 8, 32-35.

Niblack, W., Barber, R., Equitz, W., Flickner, M., Glasman, E., Petkovic, D., Yanker, P., Faloutsos, C. and Taubin, G. (1993). The QBIC project: querying images by content using color, texture, and shape. Storage and Retrieval for Image and Video Databases, 1908, 173-181.