Sonic Interactions in Virtual Environments

Preface
Acknowledgements
Contents
Editors and Contributors
Part I Introduction
1 Sonic Interactions in Virtual Environments: The Egocentric Audio Perspective of the Digital Twin
1.1 Introduction
1.2 SIVE: From an Archipelago to a Research Field
1.3 Egocentric Audio
1.3.1 Spatial Centrality
1.3.2 Entanglement HCI
1.3.3 Auditory Digital Twin
1.4 A Taxonomy for SIVE
1.4.1 Immersion
1.4.2 Coherence
1.4.3 Entanglement
1.5 Conclusion
References
Part II Interactive and Immersive Audio
2 Procedural Modeling of Interactive Sound Sources in Virtual Reality
2.1 Introduction
2.2 What to Model
2.2.1 Diegetic Sounds
2.2.2 Everyday Sounds
2.3 Perceptual and Cognitive Aspects
2.3.1 Latency, Causality, and Multisensory Integration
2.3.2 Everyday Listening and the Plausibility Illusion
2.3.3 Active Perception, Place Illusion, Embodiment
2.4 Events Versus Processes
2.4.1 Event-Driven Approaches
2.4.2 Procedural Approaches
2.4.3 Physics-Based Methods
2.4.4 Computational Costs
2.5 Procedural and Physics-Based Approaches in VR Audio
2.5.1 Methods
2.5.2 Optimizations
2.5.3 Tools
2.6 Conclusions
References
3 Interactive and Immersive Auralization
3.1 Introduction
3.2 Architecture of Real-time Auralization Systems
3.2.1 Computational Cost
3.2.2 Modular Design
3.2.3 Propagation
3.2.4 Spatialization
3.3 Mathematical Model
3.3.1 The Green's Function
3.3.2 Impulse Response
3.3.3 Directional Impulse Response
3.3.4 Bidirectional Impulse Response (BIR) and Rendering Equation
3.3.5 Band-limitation and the Diffraction Limit
3.4 Structure and Perception of the Bidirectional Impulse Response (BIR)
3.4.1 Physical Structure
3.4.2 Initial (``Direct'') Sound
3.4.3 Early Reflections
3.4.4 Late Reverberation
3.5 System Design Considerations for VR Auralization
3.5.1 Room Auralization
3.5.2 VR Auralization
3.6 Rendering the BIR: the Deterministic-Statistical Decomposition
3.6.1 Deterministic Component, Dd
3.6.2 Statistical Component, Ds
3.7 Computing the BIR
3.7.1 Geometric Acoustics (GA)
3.7.2 Wave Acoustics (WA)
3.8 Auralization Systems
3.8.1 Room Acoustics for Virtual Environments (RAVEN)
3.8.2 Wwise Spatial Audio
3.8.3 Steam Audio and Resonance Audio
3.8.4 Project Acoustics (PA)
3.9 Summary and Outlook
References
4 System-to-User and User-to-System Adaptations in Binaural Audio
4.1 Introduction
4.1.1 Localisation Cues and Their Individual Nature
4.1.2 Minimising HRTF Mismatch Between the System and the Listener
4.2 System-to-User Adaptation: HRTF Synthesis and Selection
4.2.1 HRTF Modelling
4.2.2 HRTF Selection
4.3 User-to-System Adaptation: HRTF Accommodation
4.3.1 Training Protocol Parameters
4.3.2 HRTF Accommodation Example
4.4 Discussion
4.5 Conclusions and Future Directions
References
5 Audio Quality Assessment for Virtual Reality
5.1 Introduction
5.2 Perceptual Qualities and Their Measurement
5.2.1 Generic Measures
5.2.2 VR/AR-Specifc Measures
5.2.3 VR/AR-Specific User Interfaces, Test Procedures, and Toolkits
5.3 Audio Reproduction Techniques
5.3.1 Sound Field Analysis and Synthesis
5.3.2 Binaural Synthesis
5.3.3 Binaural Reproduction of Synthesized Sound Fields
5.4 System Performance
5.4.1 Binaural Synthesis
5.4.2 Sound Field Synthesis
5.4.3 Binaural Reproduction of Synthesized Sound Fields
5.5 Conclusion
References
Part III Sonic Interactions
6 Spatial Design Considerations for Interactive Audio in Virtual Reality
6.1 Introduction
6.2 Background
6.2.1 Terminology
6.2.2 Standing on the Shoulders of Giants, but Which Ones?!
6.2.3 Typologies and Spatial Analysis
6.3 Design Analysis
6.3.1 Methodology
6.3.2 Typology of Virtual Reality Interactive Audio Systems
6.4 Spatial Design Analysis Case Studies
6.4.1 Single-User Systems
6.4.2 Collaborative Systems
6.4.3 Collective Systems
6.4.4 Spatial Audio Production Systems
6.5 Discussion and Implications
6.5.1 Spatial Design Considerations
6.5.2 Role of Space and Interaction
6.6 Research Directions and Opportunities
6.6.1 Embodied Motion Design
6.6.2 Designing for Collaborative Sound and Music in Virtual Reality
6.6.3 Spatial Audio Production for Immersive Entertainment
6.7 Conclusion
References
7 Embodied and Sonic Interactions in Virtual Environments: Tactics and Exemplars
7.1 Introduction
7.2 Soma Design
7.2.1 Defamiliarization: Making Strange
7.2.2 Perspectives
7.3 Soma Design Exemplars for Balance
7.3.1 Balance Rehabilitation
7.3.2 SWAY
7.3.3 Snap-Snap T-Shirt
7.3.4 Slow Floor
7.4 Work in Progress: Balance Rehabilitation
7.4.1 System Architecture
7.4.2 Soma Design Elements
7.4.3 Initial Observations
7.4.4 Test Procedure
7.4.5 Observations
7.5 Conclusions and Future Work
References
8 Supporting Sonic Interaction in Creative, Shared Virtual Environments
8.1 Introduction
8.2 Shared Virtual Environments
8.2.1 Embodiment in Collaborative Virtual Environments
8.2.2 Collaborative Music Making
8.3 LeMo: An SVE Supporting CMM
8.4 Study I—Visual Approach: 3D Annotation
8.4.1 Participants and Procedure
8.4.2 Annotation Categories
8.4.3 Interviews
8.4.4 Reflection of Study I
8.5 Study II—Audio Approach: Augmented Acoustic Attenuation
8.5.1 Hypotheses
8.5.2 Independent Variable
8.5.3 Dependent Variables
8.5.4 Participants and Procedure
8.5.5 Results
8.5.6 Discussion
8.6 General Discussion
8.6.1 Modality and Interaction Type
8.6.2 Key Support for Collaboration
8.6.3 Characteristic and Application
8.7 Conclusions and Future Work
References
9 Spatial Audio Mixing in Virtual Reality
9.1 Introduction
9.2 Audio-Visual Interaction
9.3 Designing Computer Music in VR
9.3.1 Technology
9.3.2 Interaction
9.3.3 Sound in Space
9.3.4 Graphical Interface
9.4 Existing Mixing Interfaces
9.5 Target Group
9.6 Conceptual Overview
9.7 Virtual Environment
9.7.1 Rendering and Lighting
9.7.2 Interaction
9.7.3 Shaders and Visual Appearance
9.7.4 Audio Design
9.7.5 Summarising Design
9.8 Implementation
9.9 Creating the Computer Version
9.10 Evaluation
9.11 Setting and Procedure
9.12 Participants
9.13 Results
9.13.1 Focus Group Interview
9.13.2 Mixing Task
9.14 General Discussion
9.15 Conclusion
References
Part IV Sonic Experiences
10 Audio in Multisensory Interactions: From Experiments to Experiences
10.1 Introduction
10.2 Audio-Visual Interactions
10.3 Embodied Interactions
10.4 Audio-Haptic Interactions
10.5 Conclusions
References
11 Immersion in Audiovisual Experiences
11.1 Introduction
11.2 Conceptualizations of Immersion
11.2.1 Psychological Perspective
11.2.2 Physical Perspective
11.3 Immersion: A Cognitive Concept
11.3.1 Quality of Experience (QoE) and Immersion
11.4 Differentiating Immersion from Interchangeably Used Terms
11.4.1 Presence
11.4.2 Flow
11.4.3 Envelopment
11.5 Subjective Assessment of Immersion: An Exploratory Study
11.5.1 Subjective and Objective Measures
11.5.2 Research Questions
11.5.3 Experimental Strategy and Design
11.5.4 Methods
11.5.5 Procedure
11.5.6 Results
11.5.7 Discussion
11.6 Summary and Future Work
References
12 Augmenting Sonic Experiences Through Haptic Feedback
12.1 Introduction
12.1.1 Multisensory Processing of Touch and Audition
12.1.2 Chapter Outline
12.2 Ball Bouncing on Everyday Materials
12.3 Reproduction of Target Pressing Forces
12.4 Vibrotactile Recognition of Traditional Musical Scales
12.5 Perception of Plucked Strings
12.6 Piano Playing
12.7 Digital Piano Playing
12.8 Playing Experience on a Haptic Surface for Musical Expression
12.8.1 Design
12.8.2 Results
12.9 Conclusions
References
13 From the Lab to the Stage: Practical Considerations on Designing Performances with Immersive Virtual Musical Instruments
13.1 Introduction
13.1.1 The Role of Scenography
13.1.2 A New Approach to IVMI Performance
13.2 Constraints of the Digital Live Music Experience
13.2.1 Stage Performance
13.2.2 Communication Between Performers and Audience
13.2.3 Music Ensemble
13.3 Virtual Environments and the Constraints of Immersive Experiences
13.3.1 Immersion
13.3.2 3D Interaction
13.3.3 Collaboration and Observation
13.4 Dimensions of IVMI Scenographies
13.4.1 Performers Transportation and Spectators Transportation
13.4.2 Spectators Awareness
13.4.3 Performers Visibility and Spectators Visibility
13.4.4 Ensemble Potential
13.4.5 Interaction Spectrum
13.5 Case Study: Analysis of IVMI Performances
13.5.1 Approaching a Performance Analysis
13.5.2 The Sound of One Hand
13.5.3 Virtual_Real
13.5.4 Drile
13.5.5 The Reggie Watts Experience
13.5.6 Resilience
13.6 Towards the Design of Novel Scenographies
13.6.1 Co-located Antithetical Immersion
13.6.2 Augmented Workspace and Spatial Paradox
13.6.3 Double-Sided Virtual World
13.7 Conclusion
References
Index