Experimental research about the correlation of sound and image in Motion Graphics YouJeong PAIK*, Mahnoo KWON**, Adam SINYKIN***, JaeMyoung KIM**** *Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608-736, KOREA, amidalapaik@hotmail.com **Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608-736, KOREA,, mahnoo@star.ks.ac.kr ***Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608-736, KOREA,, adam@star.ks.ac.kr ****Graduate School of Digital Design, KyungSung University & 110,Daeyeon-dong, Nam-gu, Pusan, 608-736, KOREA,, jmgim@star.ks.ac.kr Abstract: The purpose of this study is to highlight aspects of audio/visual synergy within the general field of motion graphic technology. Contemporary research suggests that the preeminence of image in motion graphic production is giving way to a strategic fusion of sound and image as the paramount carrier of information and emotional impact. Visual information is conveyed through a perceptual media. Audio information is conveyed through a physical medium, (the body actually resonates with incoming acoustic vibrations) and it is primarily through physical means that subliminal, emotional and linguistic information is transmitted. This was not always the received view. For years it was accepted that visual information was the signature carrier of media data and media effect. Recent data, especially IT and neuro-linguistic research, advance the notion that acoustic effects are actually more effective than information conveyed by purely visual means. Motion graphic technology is time based media with distinctive spatial characteristics, and is well suited as a laboratory for experiments in media effect with regard to sound and image. This paper will outline a series of prototypes and experiments utilizing MindSensor II software and brain wave topology interface (ECTI) in the relative effectiveness of various sound/image combinations conveyed through motion graphic media. This paper will conclude with an executive summery of the experimental data, and a review of recent developments in the synergistic effects of sound and image. Key words: Motion Graphics, Cognitive information, Physical Information 1. Introduction Motion Graphics is one of the most persuasive multimedia genres. It has the potential to fuse audio and visual stimulus more effectively than any other digital media. Within this genera, as the technology improves, audio data will become as important a carrier of information as visual data. Motion Graphic work can express powerful emotional content, exploiting the dynamic interrelation of visual kinetics and acoustic texture and rhythm.[1] As Marshall McLuhan argues, the mechanics of the media itself is as important a carrier of emotional effect as is the cognitive content.[2] Acoustic material, the prime driver of physical and emotional effects in multimedia,
will gain increasing importance in the deployment of motion graphics technology. In this paper, experimental research will focus on physical information (i.e. audio.) and it s interrelationship with a series of stable parameters of visual (i.e. cognitive) information. This study will establish an experimental methodology in order to gage the varying effects and correlation of cognitive and physical information on human brain waves. This paper will conclude with an executive summery of the experimental data, and a review of recent developments in the synergistic effects of sound and image. 2. Method The first step of this research is making audio-visual prototype samples. The samples consist 8 categories.<table1> Visual classification lend its concept from JAGDA(Japan graphic designers association) model.[3] Next step, every subjects are exposed these 4 audio-visual samples. Finally, gathering brainwave data and results were analyzed with statistic software. To make up for the weak points of quantitative analysis in-depth interview followed. The topography of cognitive activity and emotional responses to external stimuli can be studied by analyzing the variation of EEG potentials in brainwaves. Hans Berger, the discoverer of brainwaves, was the first to demonstrate that as aspects of the psyche can be measured using electronic apparatus. Berger argued that brain waves constituted a rough analog to thought and emotion.[4] This research will explore the correlation of sound and image in a scalable study of brainwaves in the context of motional graphic stimuli. Human emotions can not be error-freely measured into quantitative data because they have marked tendency of subjective aspects. Emotion can be defined as strong psychological change and inevitably accompanied with physiological symptoms.[5] Brainwave is the most salient of those symptoms. So this research tried to capture meaningful data about subjective emotion through objective detect device like brainwave measurement system. Table.1 Experiment design Audio visual Accidental Mechanical Organic Freehand Audio Sample 1 Sample 3 Sample 5 Sample 7 No audio Sample 2 Sample 4 Sample 6 Sample 8 3. Background 3-1 Motion Graphics Motion graphics refers to a synthesis of elements common to animation and traditional visual design. It is distinguished from other design media in that it moves by timeline enabling design professionals to compose images using musical and painterly techniques.[6] Motion graphics achieved preeminence in the design world in a surprisingly short period, and is now considered the privileged genre within the field of visual design. Applications for motion graphics include advertising, music video, cinema, TV, Web and DVD. With the integration of UI technology, motion graphics has become an interactive genre, incorporating hypertext and non-linear elements.[7]
3-2 Characteristic of the brain waves There are four types of brainwaves; alpha (8~13Hz), beta (14~50Hz), theta (4~7Hz), and delta (3~5Hz). Alpha and beta waves are generally referred to as fast waves and theta and delta waves as slow wave in reference to their electronic amplitude, but in general, the stimulus response of various brainwaves tend to be interrelated. Normally, alpha waves are characteristic electronic signifier in routine activities. In case of tension or excitement beta waves become dominant. During rest or relaxation and deep sleep, theta and delta waves appear.[8] Certain tasks that require high levels of attention tend to suppress alpha or non-synchronous waves and stimulate beta waves. 3-2-1 alpha waves (α) Characteristic of routine activities. Frequency range 8~12Hz and amplitude of frequency 50µV. 3-2-2 beta waves (β) Indicative of activity that requires attention or immersion. Frequency range 14~50Hz and amplitude 5~10µV. 3-3-2-3 theta waves (θ) Theta waves appear during strong excitation or shallow sleep condition. Frequency range 4~7Hz and amplitude is high and regular. 3-2-4 delta waves (δ) Delta waves are called sleeping waves. Delta waves are frequently symptomatic of trauma to the cerebrum, especially in children. Frequency range 3~5Hz and amplitude 200µv. 3-3 Physical information and cognitive information Human beings have five senses; sight, hearing, smell, touch and taste. The sense of sight and hearing are most important for operative communication in multimedia. For this reason, computer systems rely on sight and hearing, almost exclusively. Even here, the role of sound is underexploited. If computer systems or human perception lack audio stimulus, information redundancy increases.[9] UI research indicates that with enhanced deployment of audio interface, information efficiency could be maximized. The sense of sight is associated with cognitive processes. The sense of hearing is actually an emotional process because acoustic vibrations directly stimulate the human body.[10]<fig.1> Motion graphic designers utilize sound elements because they are the most important emotional element of the transmission process. However, visual and audio elements are interrelated. Fig.1 Cognitive, Physical information 4. Experiment design 4-1 laboratory setting and participants The experiment is conducted in soundproof environment. A random selection of subjects with set age and
educational parameters participated.<fig.2> Ten subjects participated pre-experiment and 10 males and 11 females participated in the main experiment.<fig.3> Participants are students of Kyungsung university. Fig.2 Soundproof walls Fig.3 subject interview 4-2 Equipment Normally medical brain wave detectors consist of more than 20 sensors and each sensor has a sucker. For this reason all participants have to shave their hairs for experiment. And the purpose of medical sensor is to catch microscopic abnormal signs of brain, for example epilepsy. So they are not fit for measuring emotional changes. But thanks to the radio frequency technology development it is possible now to record overall change pattern of brainwave with only one hair band sensor. This experiment uses a single-wide sensor system called Brain Builder Unit. The software is called Mind Sensor II, which is commonly used to gather and analyze brainwaves. Both analytical applications operate in Windows XP environment. Statistical analysis used SPSS 10.0 (Statistical packages for Social Sciences) 4-2-1 Interface: Brain Builder 4-2-2 Software: Mind SensorII 4-2-3 EEG spectrum: frequency range 1 ~ 24 Hz / pressure 1µVpp / density 0.6µV / sampling frequency 128Hz / channel variant: 1:1 EEG band is set from anterior to posterior lobe for brainwave signature. 4-3 Visual Variables In the context of contemporary perceptual and color perception research, four motion graphic animations are utilized; accidental, mechanical, organic, and freehand. Visual prototypes were classified into four groups as follow figure. <Fig. 4> Each type has opposite visual meaning by vertical and horizontal direction.
Fig.4 Visual category According to these visual classification principles four motion graphic works were made below. 4-3-1. Accidental: images without perceptible morphology. Fig.5 Accidental Image sample 4-3-2. Mechanical: refers to mathematically or mechanically generated images. Fig.6 Mechanical Image sample 4-3-3. Organic: morphology generated through natural processes/complex environmental forces.
Fig.7 Organic Image sample 4-3-4. Freehand: an image generated by human gesture(s). Fig. 8 Freehand Image sample 4-4 Audio variables The audio variables were made by arbitrary criteria because which sound is exactly matched with the visual images can not be scientifically verified. Sound matching is not a realm of science but of art. We made four types of sound samples fit for visual images using various audio samples and sound effects. The coordinates and sampling procedures are analogous to the methods used in the composition of the visual files. 4-5 Procedure The measurement of brain waves was conducted in 2 stages: Pretest and Main test. Pretest cataloged EEG characteristics and subject bias. Each participant was exposed to stimulus of varying density for trial periods of 5 minutes, forty seconds. Presentation classification included accidental, mechanical, organic, and freehand visual files. Each motion graphic animation lasted for 20 seconds. The Main Test was repeated 8 times per person. There were a total number of eight AV files. Each person was tested for 1 minute and 20 seconds per AV file. 5. Results Five categories of brainwave data were collected; Beta, fast alpha, mid alpha, slow alpha, theta waves.
Statistical analysis determines mean differences between two variables per subject/total base.(t-test and F-value test) Analysis results say that T-test between male and female has no significant differences in brain wave frequencies. And there are no statistical differences between blood types depend on F-value test. These are partially caused by small sample size of subject. To analyze effects of audio-visual types this research calculated dominant brain waves in each stimulus. Main results bellow. Each cell represents a subject who has the dominant waveform profile for each variety of AV file. (multiple response.) Table.2 Number of subject with predominant wave(audio) wave Fast Mid Slow β Stimulus α α α accidental 5 10 3 15 mechanical 8 8 5 11 organic 6 11 4 14 freehand 6 11 2 11 Table.3 Number of subject with predominant wave(no audio) wave Fast Mid Slow β Stimulus α α α accidental 3 2 2 18 mechanical 2 2 1 16 organic 4 3 3 15 freehand 4 2 3 13 According to the analysis, beta and fast alpha waves were dominant in sound added situations and slow alpha waves are distinctively notable in the silent trial. These present that every subjects exposed to sound have much more attention to stimulus. <Table2><Table3> It was evident that the subjects perceive more information when visual stimulus is accompanied by sound. Organic and Freehand files elicited minimal beta activity when projected without audio. With accompanying audio, mechanical and organic files elicited high response in beta and fast alpha waves. 6. Conclusion & Limitation of this study As a result of our experiment, we can accept the following: If there is sound, the attention of subjects is considerably higher. Among 4 kinds of visual stimuli, mechanical / organic has the most significant impact on active and high-density brain waves. In other words, if sound is merged with visual data, brain wave activity is accelerated. Each of the four visual stimuli have different wave pattern.
The variations of the 4 brain wave patterns would be the subject for continuing research. These variations could be a factor of demographics, gender, or any number of other personal or social characteristics. Further research should use standardized visual files from encoded visual genres. Standardized criteria for audio files are also central to further EEG research. Emotional effects can hardly be measured with quantitative tools, but this experiment seeks to move toward a methodology that could explore human emotions and motion graphics in the context of cognitive and experimental psychology. Such research could lead extraordinary innovations within the field of motion graphic production. Further research could open startling vistas in the convergence of art, industrial psychology and science. Effective motion graphic producing is a matter of arts rather than sciences. Superb motion graphic works was mostly made by creativity or artistic inspiration. But good designer always pursues both art and science together. Good motion graphic design also lies in the middle of arts and science. This paper tried to handle arts scientifically. And this kind of efforts can produce scientific method or predictable way to produce effective visual and sound combination someday. Reference 1. Matt Woolman, Sonic Graphics Seeing Sound, Rizzoli(2000) 2. Marshall McLuhan, Understanding Media : The Extensions of Man, McGraw Hill(1965) 3. JAGDA Hwaseon Kang trans, JAGDA textbook vol1 Visual Design, seoul : artbook(1995) 4. Tonoike, M., Yamaguchi, M., Nakayama, N. and Ikemi, Y., Brain topography study of aromacological relaxation using EEG and MEG, in Brain Topography Today, ed. by Koga, Y., Nakata, K. and Hirata, K. Elsevier, Tokyo, 733-739(1998) 5. Hyung-Gun Yun, Emotion and Emotional Engineering, Design Research Vol 3, Korea Design Society(1998) 6. Peter Hall & Andrea Codrington, PAUSE: 59 MINUTES OF MOTION GRAPHICS, Universe(2000) 7. Kathleen Ziegler, Nick Greco, Tamye Riggs, Motion Graphics Film + TV, HBI(2002) 8. Daesik Kim & Jangwook Choi, Electroencephalogram, Korea Medical Books(2001) 9. David Sonnenschein, Sound Design, Michael Wiese(2001) 10. Abeles, H.F., Responses to music, In D. A. Hodges(Ed.), Handbook of music psychology, Lawrence, KS: National Association for Music Therapy(1980)