Audio-visual perception of acoustical environments
Final Report Abstract
The effects of acoustic and optical information on various auditory, visual, and audiovisual (geometric, aesthetic, presence) features relating to the perception of and in rooms were investigated. To this end, participants were presented with music and speech performances in six rooms varying in size and material (involving acoustic absorption) and asked for their ratings. A 3D virtual environment featuring rich cue conditions allowed for the mutually independent variation of the optical and acoustic components of the artistic renditions and the spaces in which they were staged, as well as for the investigation of the effects of both the presence and the properties of the acoustic and optical domains by integrating co-presence and conflicting stimulus design paradigms. Neither crossmodal effects nor appreciable interaction effects of acoustics and optics were observed, indicating that for most of the auditory, visual, and audiovisual features the perceptual processing of acoustic and optical information is straightforward. Regarding the audiovisual features, the geometric estimates relied predominantly on seeing, the aesthetic judgments predominantly on hearing. In the second phase, certain topics such as distance perception (exp. 1), intermodal capabilities (exp. 2, 4), the effects of room acoustic parameters (exp. 3), and the perceptual evaluation of loudspeakers (exp. 7) were focused, and potential subject- and stimulus-related moderator variables were investigated (exp. 5, 6). For exp. 1, a second, smaller variant of the virtual environment was installed. New acoustic and/or optical stimuli comprising rooms and content were produced. A methodological framework for basic research into audiovisual perception was proposed. Looking at the empirical results, exp.1 did not only confirm the matter of course that both source distance and room size perception rely clearly on optical information. Room acoustics influenced the perceived distance, and the physical distance influenced the perceived room size in turn. Lowabsorbent rooms provided more perceptually exploitable cues than high-absorbent rooms leading to both a relatively greater acoustic contribution to and a greater accuracy of distance perception. Exp. 2 resulted that non-experts are generally not capable of correctly assigning optical and acoustic prototypical room shapes (cube, dome, pyramid, shoebox, theatre, vineyard). A coarse pattern of confusion was identifiable. Exp. 3 showed that the room acoustic parameters RT and C80 proved true to be determinant for important auditory, aesthetic, and geometric features of performance rooms. Construing the concept of acoustic intimacy as the perceived distance owing to a mere acoustic stimulus, RT and C80 were found to affect it considerably. An appreciable interaction of RT and C80 with the optoacoustic co-presence was observed only for geometric features. Exp. 4 on audiovisual matching of perceived and memorized rooms failed because it resulted that most of the non-expert participants were not capable of distinguishing between five acoustic rooms varying in size and acoustic absorption. Hence, participants were also not capable of correctly assigning the most of the five acoustic and optical rooms. Exp. 5 indicated that experiments differing in the optical simplification of simulated sound sources should be compared with caution. The representational veridicality of simulation matters to some extent and in a complex way. Exp. 6 showed that both the affinity to classical music and the frequency of concertgoing influenced the effects found in the first phase of the project. Geometric features were, however, not affected. The tested types of expertise generally shifted the relative importance of hearing and seeing for the auditory and aesthetic features towards hearing. Exp. 7 revealed effects of the optical presence of home speaker systems on few auditory features and an audiovisual one, as well as small crossmodal effects of the speakers’ optical properties on six auditory features. Remarkably, the monetary values ascribed to the speakers were dominated by the optical properties, not by the acoustic. The findings contribute to the understanding of the principles of multi-modal perception and broaden the basis for an empirically-founded model on audiovisual perception in and of rooms; they further indicate to what extent results of experiments conducted under acoustic stimulus conditions may be transferred to optoacoustic stimulus conditions; and they may be useful for the perceptually-based advancement of optoacoustic virtual environments and related content. Future research might specify further determinants of the perception of geometric dimensions, reveal stimulus- and subject-related factors for the perceptual distinguishability and memorizability of room acoustics and their assignability to room optics, and test the influence of present optical information on the perceptual effects of further room acoustic parameters. Besides the publication of the projects' results in scientific journals, the project was reported on by a popular scientific journal, specialized magazines, daily newspapers, and radio stations. Süddeutsche Zeitung (full-page story "Ganz Ohr", Wissen, 2-8-2016) Die Welt (front-page story "Ich sehe was, was Du nicht hörst", 19-7-2016) Neues Deutschland ("Hörer unbeirrbar", 19-7-2016) Neue Osnabrücker Zeitung ("Beim Musikhören trennen wir Klang und Ort", 19-7-2016) RBB Kulturradio (studio interview "Wie beeinflusst das Hören das Sehen – und umgekehrt?", Kulturradio am Vormittag, 19-7-2016, 9.10 a.m.) DLR Kultur (telephone interview "Wie das Sehen das Hören beeinflussen kann", Studio 9, 18-7-2016, 12.33 p.m.) DLR Kultur (segment "Das Auge hört mit", Kompressor, 20-7-2016, 2.47 p.m.)
