Multimedia Database Management Systems (MMDBMS): An overview of design, retrieval methods, and applications. Vanna Bushong. Evangel University

Transcription

1 Multimedia Database 1 Running head: MULTIMEDIA DATABASE MANAGEMENT SYSTEMS Multimedia Database Management Systems (MMDBMS): An overview of design, retrieval methods, and applications Vanna Bushong Evangel University

2 Multimedia Database 2 Introduction to Multimedia Database Management Systems (MMDBMS) Search the hard drive of the average user s computer, and you will most likely find pictures, music, and videos taking up a large amount of space. Media is everywhere, and people are collecting more of it as the years go by. With the popularity of YouTube videos and Facebook photo albums, for example, the focus in multimedia research has shifted from creating multimedia to storing and retrieving it. About forty years ago, when more and more information began to be stored as computerreadable data, a system was developed to make storage and retrieval of this information a simpler process. The system was called a Database Management System (DBMS), and it is used in some form in nearly every business and organization today. DBMSs use information retrieval (IR) systems, also known as text-based retrieval, to search databases for documents relevant to the user s query. This system works well for alphanumeric data, but it is insufficient for multimedia data. Another system a multimedia database management system (MMDBMS) is required to handle the storage of and access to multimedia data. (Lu, 1999) Multimedia database management is a broad topic, so the following research focuses on three important concepts. The first section covers the basic principles of MMDB design, including the architecture of a multimedia retrieval system and a general multimedia data model. The second section provides an overview of retrieval methods for audio, image, and video data. The third and final section briefly explains how MMDBMSs are used today and their potential for use in the future. Multimedia Database Design As mentioned in the introduction, text-based retrieval is not effective for multimedia data. Instead, multimedia information requires content-based retrieval, in which retrieval is based on queries of actual features, like colors and shapes. A multimedia indexing and retrieval system (MIRS) combines

3 Multimedia Database 3 DBMS techniques with IR and content-based retrieval to form a multimedia DBMS (MMDBMS). (Lu, 1999) Since multimedia data is rich in information, MIRS architecture must be flexible and expandable to handle the various features. Another factor to keep in mind is that an MIRS is usually distributed. This is because data like video and images are so large that it would not be efficient to make a copy of the database for each user. Also, multimedia data is often accessed by many users. For example, the itunes music store provides thousands of songs online for users all over the world to download. The basic architecture of an MIRS can be seen in Figure 1. (Lu, 1999) User interface Feature extractor Communication manager Communication manager Indexing and search engine Storage manager Figure 1: Architecture of an MIRS by Guojun Lu (1999)

4 Multimedia Database 4 The MIRS architecture handles the insertion and retrieval of new multimedia items into the database. According to this architecture, the user specifies an item to insert or retrieve through the user interface. A multimedia item could be something like a picture stored as a file or a video clip input from a digital video camera. The feature extractor extracts the contents of the multimedia item. For a picture, this would be data such as size, date, and color information. Then the picture and its features would be sent to the server(s) through the communication managers (there is one for each client and server). Upon reaching the servers, if the picture was sent for retrieval, the indexing and search engine searches the database for pictures that best match the features of the query. The matches are retrieved from the storage manager and sent back to the user interface via the communication managers. If the picture was sent for insertion, the indexing and search engine inserts the features into the storage manager. This architecture is basic, and it may require other managers depending on the nature of the database in practice. (Lu, 1999) In dealing with a MMDBMS, users and designers need a framework for inserting, deleting, modifying, defining, and searching multimedia items and properties in the database. A data model provides this framework, allowing efficient storage and search of the multimedia database. A general multimedia data model is shown in Figure 2. (Lu, 1999) The data model consists of three layers: the object layer, the media type layer, and the media format layer. The object, a video clip for instance, is a composite of spatial and temporal relationships, as shown in the object layer. A spatial relationship is the size and position of each object, while a temporal relationship is the object's specification in time. The media type layer includes different attributes of the media object, such as text, image, graphics, audio, and video. These types are further broken down in the media format layer. For example, the image type could be stored as a raw or compressed file, in grayscale or color, JPEG or JBIG format, etc. The information in the media format layer is used for proper decoding and presentation of the media object. (Lu, 1999)

5 Media format layer Multimedia Database 5 Object layer Spatial Temporal Composite Media type layer Text Image Graphics Audio Video Raw Compressed Grayscale Color JPEG JBIG DPCM Figure 2: A general multimedia data model by Guojun Lu (1999) As previously stated, this is a general data model that should not be considered an exact standard for every database application. The model should be adapted for the specific database to be designed, whether it is for video clips, music, slideshows, or any other media objects. Each of these objects have different features, so the model must be expanded to account for all additional requirements. Overview of Multimedia Data Retrieval Methods Retrieving data from a traditional database is a matter of comparing characters and/or numbers in a query to data in the stored records. For example, in a table of student records called STUDENT, the following SQL query would be used to retrieve the name of the student with an ID number of 25:

6 Multimedia Database 6 SELECT NAME FROM STUDENT WHERE ID#=25; When it comes to multimedia databases, however, text-based retrieval does not apply to the many nontext features of media objects, such as sounds and shapes. This section provides an overview of the content-based retrieval methods for audio, image, and video data objects, as these are three of the most common forms of media stored today. Imagine humming a tune into the microphone of your computer, and the name of the song and all the information about it appears on the screen. With content-based audio retrieval, this is possible, and very practical - since most people will remember the tune of a song before remembering its title or artist. In order to index and retrieve audio pieces, Guojun Lu suggests this approach: Audio is classified into some common types of audio such as speech, music, and noise. Different audio types are processed and indexed in different ways. For example, if the audio type is speech, speech recognition is applied and the speech is indexed based on recognized words. Query audio pieces are similarly classified, processed, and indexed. Audio pieces are retrieved based on similarity between the query index and the audio index in the database. (1999) To compare the features of one audio clip to another, the database must store samples of the audio and its characteristics, such as amplitude and frequency. Digital sound is sampled every nth microsecond, and each sample contains information which represents the sound. Typically there are thousands of samples per second, and the more frequent the sampling, the more closely the represented sound will match the original. Figure 3 on the following page shows a diagram of a sound wave that is sampled and reconstructed. (Khoshafian and Baker, 1996)

7 Multimedia Database 7 Figure 3: Sound waveforms and sampling (Khoshafian and Baker, 1996) The information collected at each sample point can be used to determine the characteristics of the audio clip. These characteristics include bandwidth, spectral centroid, silence ratio, zero-crossing rate, and regular beat - all topics which are beyond the scope of this paper (Lu, 1999). Suffice it to say that if an audio clip is used as a query, a sampling procedure is performed on this object. Then the indexing and search engine of the MMDBMS uses its characteristics to compare with those already in the database. Music, for example, has a higher frequency range than speech. So a clip with a frequency in the range of music could be categorized as such, then additional algorithms could compare more features until a matching song is found. Image databases are another popular form of multimedia databases which are more effective to use with content-based retrieval. Pictures could be saved in a database along with keywords to describe them. The database could also store attributes of each picture such as image name, date taken, type of camera, author, etc. So text queries like "children" and "January 7, 2003" would return pictures of children and pictures taken on January 7, There are some obvious limitations to text-based retrieval of images - for one, it is often difficult to describe a picture in words. Therefore, two content-

8 Multimedia Database 8 based methods have been developed, feature extraction/object recognition and low-level image features. Feature extraction and object recognition is currently not practical and is still being researched, but retrieval based on low-level image features, such as color, shape, and texture, is very common. (Lu, 1999) Color-based retrieval is the most common content-based method for retrieving pictures, so it is the method I will focus on in this section. A color is made up of three different values of the primary colors: red, green, and blue (RGB). For each pixel in a digital image, the computer records the RGB values and maps the color to a cell in a histogram. A histogram is a visual graph showing an image's color distribution, or the number of pixels of each color. By comparing histogram values, you can find images with similar color distributions. Color-based retrieval does not guarantee a perfect match, of course. But when used in combination with shape, texture, and text-based retrieval, the results can be pretty impressive. (Khoshafian and Baker, 1996) Retrieval of video clips is the final topic of this section. Video presents a unique challenge because it is a combination of audio and images over time. Because of its multiple characteristics, it makes sense to use a combination of retrieval methods for best results. The simplest techniques include the text-based and metadata-based methods. Text retrieval could be performed using the video's subtitles, and metadata information such as title, director, and date of release could also be indexed and searched. (Lu, 1999) The more complicated methods of video retrieval include the audio-based and content-based methods. Audio content from a video clip can be categorized as music, speech, or other sounds through the sampling procedure previously mentioned. The MMDBMS can then store the audio characteristics associated with each frame of video and use them for comparisons. Video content retrieval begins by segmenting the video into frames, shots, and scenes. A frame is a single still image within a video, while a shot is collection of frames that focus on one object or

9 Multimedia Database 9 event. A scene is series of shots in one general location. Segmenting the video in this manner creates a collection of smaller storyboards which represent the entire video. (Bashir and Khokhar, 2003) Once the video has been segmented, each shot is usually represented by a key frame. A key frame captures the essence of the shot in regards to color, shape, etc. Key frames can be indexed and retrieved just like images. The drawback to this approach is that it treats video like still photos and ignores motion and changes in images over time. (Lu, 1999) To solve the problem of lost motion characteristics in a video database, some object-based indexing schemes have been proposed. In this type of system, pixels of the video that "move" together are grouped into an object, and the video is indexed based on these objects. It is beyond the scope of this paper to describe the exact process by which pixels are determined to be similar and moving together. However, it should be noted that object-based retrieval is easy to implement when the video is compressed into an MPEG-4 format, due to the fact that objects are specifically represented in the bitstream in this format. (Lu, 1999) Figure 4 on the following page is an excellent model of a video indexing and retrieval system. As shown, the MPEG-4 format breaks the video file into a video and audio bitstream. The video stream is further broken into video objects, and the features of each object are stored in the database, along with the audio features. When users search the database, the features they use as queries are compared with features in the indexed database. The best matches are then displayed to the user interface. (Bashir and Khokhar, 2003)

10 Multimedia Database 10 Figure 4: A hypothetical Video indexing and retrieval system operating on raw video or MPEG-4 compressed bitstream using multimedia features for indexing and retrieval. (Bashir and Khokhar, 2003) Indexing and retrieving content from a multimedia database is no simple task. As this section has demonstrated, there are numerous techniques for finding a good match between the user's query and the information stored in the database. Multimedia retrieval is a broad subject and this paper has only skimmed the surface of a few common techniques for sounds, images, and videos, but this is a good starting point for understanding the process.

11 Multimedia Database 11 Applications of Multimedia Database Management Systems Multimedia database management is a relatively new field, but it is bringing a trend that is undoubtedly irreversible (Khoshafian and Baker, 1996). Everyone seems to be using multimedia today, so having a system to efficiently store and retrieve it improves nearly every user's experience with computers. Multimedia databases already have numerous uses, and they hold even greater possibilities for the future. Current applications of multimedia databases are all around us. If you have ever used the speech recognition feature on your computer, you have already used an audio database. The internet provides a gigantic database of photos that you can search through keywords on Google. Police use fingerprint scanning and face recognition software for keeping records of suspects. Doctors have countless uses for image databases - an ultrasound image, for example, could be scanned into a database to find similar cases of a problem (Lu, 1999). All of these uses have been developed as the technology for multimedia databases has improved. As we look to the future, imagine the potential for multimedia database applications in more everyday life situations. Children in elementary schools could visit their libraries and scan an image of an animal for a research project. The database could retrieve information about the animal - pictures, videos, and sounds - along with a list of books they could check out from the library. A college music student wants to find songs with a similar sound to that of his favorite composer, so he plays an audio clip of the song and the database returns matches for him to choose from. Applications like these are possible, and could become commonplace in schools in the future. There are endless possibilities for multimedia database applications in the fields of education, science, entertainment, medicine, and more. (Lu, 1999)

12 Multimedia Database 12 References Bashir, Faisal I. and Ashfaq A. Khokhar. Video Content Modeling: An Overview. In International Workshop on Frontiers of Information Technology, Khoshafian, Setrag and A. Brad Baker. (1996). MultiMedia and Imaging Databases. San Francisco: Morgan Kaufmann Publishers, Inc. Lu, Guojun. (1999). Multimedia Database Management Systems. Boston: Artech House.