Perceptual content description language and interface for supporting visual semantic computing

The rapid growth of visual data has attracted a significant research interest in cognition science, computer vision, information retrieval, databases and machine learning to develop new methods for handling large image and video databases and visual data streams based on its semantic content. The field of Visual Semantic Computing (VSC) brings together those disciplines concerned with connecting human intention (often vaguely-formulated) with images and video. The users of various visual information systems may perceive the data in different ways, but a common demand is the ability to process and query the data based on their own perception of the data. Designing systems that understand images well enough to enable effective operations, such as search of large databases remains a very challenging problem. E.g., an image retrieval system is more than an image similarity engine. In addition to image matching, it should address the problems of indexing to enable fast searches; accounting for prior information, which can be used to weigh some images more strongly than others; and exploring the user's presence in the retrieval loop. Despite the sustained efforts in recent years, the paramount challenge still remains on bridging the semantic gap between human's perception on visual data and the features seen by the computer. The goal of this research is to tackle this challenge in a symmetrical way emphasizing perception based content representation and manipulation. 1) Investigate the key theories and hypotheses from human vision and cognition research in visual attention, modularity, perception, and memory. 2) Duplicate some of the key functionalities in coherent and collaborative way for extracting perceptual features, in particular Perceptual Partitioning and Grouping based. 3) Develop a frame work of Perceptual Content Description Language (PCDL) based on 1) and 2). 4) Develop a systematic approach for constructing Visual Semantic Taxonomy (VST) and mapping between VST and PCDL in that the perceptual iconic based descriptions are linked to semantic words in prior. This new approach will facilitate VSC application development for content based image/video retrieval, indexing, annotation, manipulation and recognition.