Chapter 4
Video and Image Content
Representation and Retrieval

Here we highlight some areas and systems created for the following purposes:

Querying and retrieval in image information systems:

Pictoral SQL (PSQL) -- extension of SQL which supports user-defined abstract data types for definition of pictoral domains. Spatial operators are introduced.
Query by Pictoral Example (QPE) -- similar to QBE. Another similar language - PICQUERY.
PROBE -- query that supports point sets with a geometry filter.
Intelligent Image Database System (IIDMS): queries by name, keywords, frame number, iconic example, 2D strings.
VIMSYS project -- uses interactive graphical interface for queries.
Active Index system -- raw image data can be accompanied with active index cells that are capable of communicating among themselves and invoking actions. Result -- dynamically changing index and smart images.
Image indexing algorithm by Gong and others can index images by features like color, shape (circularity), axis rotation, location of regions, histogramm content.
QBIC system by IBM features queries by example images, user constructed scetches, color and texture patterns, camera motion (frames, not objects).

Object Oriented Video Information Database (OVID) -- "interval-inclusion based inheritance". No predefined class hierarchy.Attribute values to video objects are assigned manually.
System by Little et al.- content-based video retrieval and playback.Schema composed of movie scene and actor relations with a fixed set of attributes.
Media Streams -- visual language for iconic annotations to video content. Iconoc primitives vocabulary, can be extended using combination of them.
Cut detection filter by Otsuji and Tonomura.
Salient video still by Teodosio -- extracting a representative image. Also panoramic overview from sequence of images.
Swanberg offered methods for video analyzation with image processing routines to identify the key features before inserting the video to a database.
JPEG (Joint Picture Expert Group) video compressing techniques -- compression at near real time.Operations with semi-compressed data to obtain special effects such as scaling, rotation, translation.
CMU is working on automatic mechanisms for populating a video library, for instance video skim -- short synopsis of the actual video.
Sphinx-II extracts text from audio track. Then one can extract keywords and save them with video.

Process of motion analysis is divided in 3 stages:

Related works:

Moving light displays (Johansson) -- lights attached to human joints. Audience recognizes object as a human being as well as the actions that are performed or events taking place.
Spatiotemporal surfaces (Allmen) -- projections of contours over time. Direct representation of object motion.
EMo system -- generates control functions for mechanisms that express specific emotions.

Key challenge: to find an appropriate logical representation for the entities of interest.

Most popular -- Object oriented approach -- emphasizes the interrelationships between entities and hides implementation details.
Description Based Media Object Data Model (DEMOM) -- uniform framework for different types of media data: text, images, sound, graphics.