An Image Thesaurus for Content Based Search Using Texture and Color

Bangalore S Manjunath

Electrical and Computer Engineering Department
University of California
Santa Barbara, CA 93106-9560

Contact Information

B. S. Manjunath
Electrical and Computer Engineering Department
University of California
Santa Barbara, CA 93106-9560
Phone: (805) 893-7112
Fax : (805) 893-3262
Email: manj@ece.ucsb.edu

WWW PAGE

http://vivaldi.ece.ucsb.edu/Manjunath/NSFIDM98

Keywords

image databases, visual thesaurus, texture, content based search

Project Award Information

Award Number: NSF IRI-9704785
Duration: 3 years; Project is in the first year.
Title: An Image Thesaurus for Content Based Search Using Texture and Color

Project Summary

The main goal of this project is to develop an image thesaurus to facilitate content based search of large digital image/video libraries. As computers and communications technology advance, limitations of current information processing tools are becoming more apparent, and is particularly visible in the context of media rich data such as satellite and medical imagery databases. The image or visual thesaurus provides a conceptual framework for similarity search and indexing of such multimedia data. At the time of data ingest, image features of interest, such as color, texture, spatial location and shape, are computed and clustered to give a feature code book. Application specific information and user requirements are used to make this code book construction adaptive. The mapping to code words is hierarchical and groupings of several basic features can be used to represent high level structured visual concepts. Key components in this thesaurus construction include new spatio-temporal segmentation algorithms, and dimensionality reduction and learning algorithms for efficient search and indexing in the feature space. Potential applications of this thesaurus include geographic information systems, medical imagery, searching on the Internet using image content, and in new image/video coding standards such as MPEG-7. A prototype retrieval system incorporating an image thesaurus will be developed and its effectiveness will be demonstrated on searching airphoto databases and stock photo galleries.

Goals, Objectives, and Targeted Activities

During the first year, our primary focus will be on the following research activities:

* Investigation of a rotation invariant texture feature set for similarity search in images.

* Codebook construction for airphotos

* Automated image segmentation schemes for localizing image

A prototype seach engine is being developed and tested on airphotos and color pictures of natural scenes from a Corel photo gallery.

Indication of Success

We have been quite successful in our efforts to develop the visual thesuarus. A preliminary version of the airphoto search engine has been deployed on the web. Our image segmentation scheme has yieled very promising results on a diverse collection of images from stock photo libraries.

During the summer of 1997, the NSF technology exhibits center features our demonstrations of

a texture dictionary for airphoto search, and
NeTra: region based search of color images using texture, color, and shape.

One of the objectives described in the proposal is the development of a rotation invariant texture feature set. The initial work on this is almost complete, resulting in a Masters thesis (student: George Haley). A journal paper on the same topic was recently (December 1997) accepteed for publication in the prestigious IEEE Transactions on Image Processing.

Project Impact

Human Resources

Currently one Ph.D. student is working on the project. In addition, two other students are working on related projects on digital watermarking and learning algorithms for database search.

One Ph.D. student (Wei-Ying Ma) graduated with a PhD in June 1997. His thesis was on content based search of images. He is currently employed at HP labs in Palo Alto.

Another student, Morris Beatty, is completing his MS thesis on dimensionality reduction of image features. He is now at SUN Microsystems and is expected to submit his thesis in June.

George Haley, who did some of the early work on rotation invariant texture features, completed his MS thesis last year. He is now with Rockwell, Chicago.

Education and curriculum development at all levels.

In a closely related activity, the PI is participating in the development of a Multimedia track for the Undergraduates in the ECE department at UCSB.
In addition, a new graduate program in Multimedia Arts and Technology is being proposed at UCSB. The PI is one of committee members participating in developing this new program.

Your department/institution infrastructure

UCSB is one of the six digital library sites funded by NSF and the current project builds upon and complements some of the work related to the UCSB Alexandria project. The aerial photographs required for this project have already been digitized at the Maps and Imagery Lab.

Industry -- collaborations, transfer of technology, patents.

1. A UC MICRO award ($18,000) was received for implementing some of the image processing operations on embedded DSP systems. This is in collaboration with a local company, Spectron Microsystems. Spectron was recently acquired by the Texas Instruments.

2. A UC patent is pending on our image segmentation scheme.

3. The image segmentation algorithm has been distributed on request to other research institutions. These include City University of New York, Stanford, UC San Diego, NIMA, ONR/China Lake Weapons Center, and JPL.

Project References

1. B. S. Manjunath and W. Y. Ma, "Texture features for browsing and retrieval of image data," IEEE Transactions on Pattern Analysis and Machine Intelligence, (Special Issue on Digital Libraries), Vol. 18 (8), August 1996, pp. 837-842.

2. G. M. Haley and B. S. Manjunath, "Rotation invariant texture classification using a complete space-frequency model," to appear in IEEE Transactions on Image Processing, 1998. See also G. M. Haley and B. S. Manjunath, "Rotation-invariant texture classification using modified Gabor filters," Proc. second international conference on image processing, ICIP 95, Vol. I, October 1995, pp. 262-265.

3. W. Y. Ma and B. S. Manjunath, "NETRA: A Toolbox for navigating large image databases," accepted for publciation in the ACM Multimedia Journal (1998). See also W. Y. Ma and B. S. Manjunath, "NETRA: A toolbox for navigating large image databases," Proc. IEEE International Conference on Image Processing, Santa Barbara, California, October 1997.

4. M. Beatty and B. S. Manjunath, "Dimensionality reduction using multidimensional scaling for image search," to appear in Proc. IEEE International Conference on Image Processing, Santa Barbara, California, October 1997.

5. .W. Y. Ma and B. S. Manjunath, "Texture features and learning similarity," Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, San Francisco, CA, pp. 425-430, June, 1996.

6.B. S. Manjunath and W. Y. Ma, "Browsing large satellite and aerial photographs," Proc. third IEEE international conference on image processing, ICIP 96, Vol. II, pp. 765-768, Laussnne, Switzerland, September 1996 (invited paper).

Area Background

Content based retrieval is about developing tools for intelligent browsing of the data. In traditional alpha-numeric databases, such as an on-line library catalog, we search using keywords, author names, or book titles. Similarly, generic image attributes useful for search include color, histogram, texture, and shape. However, research on content-based image retrieval is still in its very early stages.

In our initial work, we have explored the use of image texture for search and retrieval. Examples of texture images include photographs of water, sand, brick wall, a wire fence, or aerial photographs of agricultural regions. Textured images, in general, are hard to describe (i.e., they do not have good structure). Often, the resolution and distribution of objects in the scene determines if it is "textured" or not. For example, consider a bunch of coffee beans spread on the ground. While each bean is clearly an identifiable object, the random distribution of the beans as a whole is more like a texture pattern. Natural textures tend to be more irregular than man made ones. During the past six months, we have made considerable progress in developing algorithms for texture based search. The basic idea is to pre-process the images at the time of storage and extract the texture information. This is done using Gabor filters, which are modulated Gaussians. Processing through a bank of these Gabor filters is (approximately) equivalent to extracting line edges and bars in the images, at different scales and orientations.

Simple statistical moments such as the mean and standard deviation of the filtered outputs can now be used as indices to search the database. For online demosntartions of this as well as the use of color and texture to search images, see

http://vivaldi.ece.ucsb.edu/Airphoto
http://vivaldi.ece.ucsb.edu/Netra

Area References

Spcial Issue on Ditial Libraries in the IEEE Transactions on Pattern Recognition and Machine Intelligence, August 1996.

Special Issue on Digital Libraries in the IEEE Computer Magazine, May 1996.

Potential Related Projects

If you have ideas for projects or collaborations with those related program areas within the IDM, describe some of them briefly here.