Springer Handbook of Augmented Reality

Foreword
Foreword
Foreword
Preface
Contents
About the Editors
Contributors
Part I Historical Developments
1 Fundamentals of All the Realities: Virtual, Augmented, Mediated, Multimediated, and Beyond
1.1 What Is (Augmented) Reality?
1.2 Historical Background and Context
1.2.1 A Confusing Mess of Different Realities: Virtual, Augmented, Mixed, and X-Reality
1.2.2 Mediated Reality (XY-Reality)
1.2.3 Deliberately Mediated Reality
1.2.4 Unintentionally Mediated Reality
1.2.5 The Mediated Reality (X,Y) Continuum
1.3 Multimediated Reality
1.3.1 Technologies for Sensory Attenuation
1.3.2 Multimedia in Photographic Darkrooms
1.3.3 Multimediated Reality Darkroom
1.3.4 Comparison with Existing Measuring Instruments
1.4 Multimediated Reality Is Multiscale, Multimodal, Multisensory, Multiveillant, and Multidimensional
1.4.1 Multisensory Synthetic Synesthesia
1.4.2 Multidimensional Multimediated Reality
1.5 Multimediated Reality Continuum
1.5.1 Multimediated Reality is ``*R'' (All R)
1.6 Other Forms of Phenomenological Augmented Reality
1.6.1 History of the SWIM
1.6.2 SWIM Principle of Operation
1.6.3 Visualizing Radio Waves with SWIM and Software-Defined Radio
1.6.4 Electric Machines and SWIM
1.7 Summary and Conclusions
References
2 History of Augmented Reality
2.1 AR Basics: Operational Definition and Enabling Technologies
2.2 AR Enabling Technologies
2.2.1 Positional Tracking for AR
2.2.2 Co-registration and Rendering of Visual Contents
2.2.3 Visualization
2.3 The Past of AR
2.4 Current AR Technologies and Applications
2.4.1 Main Application Fields
2.5 Emerging Trends and Open Challenges for AR
2.6 Conclusions
References
Part II Principles and Fundamentals, Software Techniques, and Developments
3 Principles of Object Tracking and Mapping
3.1 Pose Estimation and Tracking Fundamentals
3.1.1 Notation: Problem Formulation
Coordinate System Transformations
6DoF Pose of Rigid Bodies
3.1.2 Cameras
Perspective Camera Model
Intrinsic Parameters
Image Distortion
Extrinsic Parameters
From World to Pixel Coordinates
Camera Calibration
Spherical Camera Model
Rolling Shutter Effects
3.1.3 Inertial Sensors
Gyroscopes
Accelerometers
Inertial Navigation
3.1.4 Depth Sensors
3.1.5 Geospatial Navigation Sensors
3.2 Computer Vision Techniques
3.2.1 Feature Matching
3.2.2 Feature Tracking
Optical Flow Estimation
3.2.3 Pose Estimation from Feature Correspondences
The DLT Algorithm
Homography Pose Estimation
Nonlinear Optimization
Random Sample Consensus
3.2.4 The Epipolar Constraint
3.2.5 Line Tracking
3.2.6 Direct Image Alignment
3.2.7 Structure from Motion
Triangulation
Bundle Adjustment
3.2.8 Depth Image Tracking/ICP
3.2.9 Deep Learning in Computer Vision
Convolutional Neural Networks
Network Training
Deep Learning in Tracking and Mapping
3.3 Model-Based Tracking
3.3.1 Marker Trackers
3.3.2 3D Model Trackers
Object Model Acquisition
Feature Trackers
Edge Trackers
Direct Trackers
Deep Learning-Based Trackers
Hybrid Trackers
Tracking by Detection/Pose Initialization
3.3.3 Nonrigid and Articulated Objects
3.4 SLAM
3.4.1 Visual SLAM
SLAM Architecture/Main Components
Keypoint-Based SLAM
Direct SLAM
3.4.2 Visual-Inertial SLAM
3.4.3 RGB-D SLAM
3.4.4 Deep Learning for SLAM
Towards Semantic SLAM
3.5 Conclusion
References
4 3D Object and Hand Pose Estimation
4.1 3D Object and Hand Pose Estimation for Augmented Reality
4.2 Formalization
4.3 Challenges of 3D Pose Estimation Using Computer Vision
4.4 Early Approaches to 3D Pose Estimation and Their Limits
4.5 Machine Learning and Deep Learning
4.6 Datasets
4.6.1 Datasets for Object Pose Estimation
4.6.2 Datasets for Hand Pose Estimation
4.6.3 Datasets for Object and Hand Pose Estimation
4.6.4 Metrics
4.7 Modern Approaches to 3D Object Pose Estimation
4.7.1 BB8
4.7.2 SSD-6D
4.7.3 YOLO-6D
4.7.4 PoseCNN
4.7.5 DeepIM
4.7.6 Augmented Autoencoders
4.7.7 Robustness to Partial Occlusions: Oberweger's Method, Segmentation-Driven MeThod, PVNet
4.7.8 DPOD and Pix2Pose
4.7.9 Discussion
4.8 3D Pose Estimation for Object Categories
4.9 3D Hand Pose Estimation from Depth Maps
4.9.1 DeepPrior++
4.9.2 V2V-PoseNet
4.9.3 A2J
4.9.4 Discussion
4.10 3D Hand Pose Estimation from an RGB Image
4.10.1 Zimmerman's Method
4.10.2 Iqbal's Method
4.10.3 GANerated Hands
4.10.4 3D Hand Shape and Pose Estimation: Ge's and Boukhayma's Methods
4.10.5 Implementation in MediaPipe
4.10.6 Manipulating Virtual Objects
4.11 3D Object+Hand Pose Estimation
4.11.1 ObMan and HOPS-Net
4.11.2 H+O
4.11.3 HOnnotate
4.12 The Future of 3D Object and Hand Pose Estimation
References
5 Mixed Reality Interaction Techniques
5.1 Introduction
5.2 Tangible and Surface-Based Interaction
5.3 Gesture-Based Interaction
5.4 Pen-Based Interaction
5.5 Gaze-Based Interaction
5.6 Haptic Interaction
5.7 Multimodal Interaction
5.8 Multi-Display Interaction
5.9 Interaction Using Keyboard and Mouse
5.10 Virtual Agents
5.11 Summary and Outlook
References
6 Interaction with AI-Controlled Characters in AR Worlds
6.1 Populating AR Worlds with Virtual Creatures
6.1.1 AI Characters in AR Literature
6.2 Designing AI Characters for AI Worlds
6.2.1 Categorization of AI Characters
Trainer, Trainee, and Coaches
Subject of an Examination
Assistants and Companions
Enemies and Opponent
6.2.2 Architecture Base Components
6.2.3 Appearance
Human-Human Communication
Human-AI Communication
Conclusions for AR Characters
6.2.4 Movement
6.2.5 Reasoning
Decision Trees
Finite State Machines
Goal-Oriented Behavior
Utility AI
6.3 Conclusion
References
7 Privacy and Security Issues and Solutions for Mixed Reality Applications
7.1 The Mixed Reality Present
7.1.1 Overview on Mixed Reality Processing
7.1.2 Towards MR Mobility
7.2 Security and Privacy Risks with Mixed Reality
7.2.1 Risks with MR Data Processing
7.2.2 Mobility and Privacy
7.3 Protection Approaches for Mixed Reality
7.3.1 Input Protection
7.3.2 Output Protection
7.3.3 Protecting User Interactions
7.3.4 Device Protection
7.3.5 Open Research Challenges
7.3.6 Future Directions
7.4 Towards Everyday MR Services
References
Part III Hardware and Peripherals
8 The Optics of Augmented Reality Displays
8.1 Introduction to Augmented and Virtual Reality
8.2 A Brief History of AR Displays
8.3 The Basics of Visual Instrument Design
8.3.1 The Human Visual System
8.3.2 Optical Design Properties for AR Displays
8.4 Optical Components of an AR Display
8.4.1 Microdisplays as the Light Engine
8.4.2 Radiometric Brightness Analysis for AR Displays to Guide Microdisplay Specifications
8.4.3 A Brief Foray into Laser Scanning
8.4.4 Imaging Optics and Combiners
8.5 Optical Architectures and How They Work
8.6 Areas for Improvement
8.7 Components and Techniques for AR Displays of the Future
8.7.1 Pupil-Steering and Eye-Tracking
8.7.2 Freeform Optics
8.7.3 Metasurfaces
8.8 Conclusion
References
9 Tracking Systems for Augmented Reality
9.1 Introduction
9.2 Multisensor Integration
9.3 Calibration Methods
9.3.1 Notations
9.3.2 Tip Tool Calibration
9.3.3 Tracking the Same Target
9.3.4 Hand–Eye Calibration
9.3.5 Absolute Orientation
9.4 Registration Methods
9.4.1 Iterative Closest Point
9.4.2 Point-Feature-Based Alignment
9.5 Inertial Measurement Unit Calibration
9.5.1 IMU Bias
9.5.2 Sensor Fusion
9.5.3 IMU–Camera Calibration
9.6 Projector–Camera Calibration
9.6.1 Pixel Mapping
9.6.2 Spatial Calibration
9.7 Optical See-Through Head-Mounted Display Calibration
9.7.1 Interaction-Based Methods
9.7.2 Interaction-Free Calibration
9.7.3 Eye Tracking
9.8 Evaluation Methods
9.8.1 Objective Measurements
9.8.2 Subjective Measurements
9.9 Tracking Systems for Sensor Integration
9.9.1 Mechanical Links
9.9.2 Electromagnetic Sensors
9.9.3 Inertial Measurement Units
9.9.4 Flex Sensors
9.9.5 Radio Signals
9.9.6 Camera-Based Motion Capture Systems
Passive Markers
Active Markers
9.9.7 Markerless Tracking
Human Pose Tracking
Facial Tracking
Hand Tracking
Thermal Tracking
9.9.8 Projection-Based Sensing
Spatial Division Code
Spatial Scanning
9.10 Applications
9.10.1 Medical Applications
9.10.2 Robotic Applications
9.10.3 Entertainment Applications
9.11 Conclusion
References
10 Embodied Interaction on Constrained Interfaces for Augmented Reality
10.1 Resurgence of Wearable Computers
10.1.1 Interaction with Today's Wearable AR Headsets
10.1.2 Drawing a Parallel to Desktops and Smartphones
10.1.3 The Constrained Interfaces on Wearable ARHeadsets
10.1.4 Rethinking on the Constrained AR Interfaces
10.1.5 Spotlights of the Chapter
10.1.6 Structure
10.2 Related Work
10.2.1 Freehand Pointing on Constrained Hardware
10.2.2 Text Entry on Constrained Screen Real Estate
10.2.3 Optimized Text Entry Layout Design
10.2.4 Summary
10.3 TiPoint
10.3.1 System Requirements
10.3.2 Interaction Overview
10.3.3 Interaction Approaches
Freehand Mode
Fast-Repetitive Mode
10.3.4 Mid-air Interaction Strategy for Small-Screen Display
10.3.5 Implementation
10.3.6 System-Wide Implementation
10.3.7 Application Scenarios
10.4 HIBEY
10.4.1 System Design
Character Keys
The Gesture of a Pointing Hand
10.4.2 Uncertainty on Keyboard-Less Environment
Probabilistic Method for Handling Imprecision
10.4.3 Implementation and User Performance
10.4.4 Application Scenarios
10.5 TOFI
10.5.1 System Implementation
10.5.2 Optimization of the Keyboard Layout
Maximizing the Goodness of Character Pair
Maximizing the Familiarity with the QWERTY Layout
Maximizing the Easiness of Force Keypad Interaction
Maximizing the Comfort Level of the Finger Space
Optimized Keyboard Layouts
10.5.3 User Performance
10.5.4 Application Scenarios
10.6 Take-Home Message for AR Interaction Techniques
10.6.1 Conclusions
10.6.2 Future Outlook: Toward the Miniature and Subtle Interfaces
References
11 Networking and Cyber Foraging for Mobile Augmented Reality
11.1 Mobile Augmented Reality Requirements and Cyber-Foraging
11.1.1 Requirements of MAR Applications
11.1.2 Network Capabilities and MAR
11.1.3 Cyber-Foraging for AR
11.1.4 Arising Challenges
11.1.5 Performance Models
11.1.6 Highlights of This Chapter
11.2 Related Works
11.2.1 Mobile Augmented Reality
11.2.2 Generic Cyber-Foraging Systems
11.2.3 Cyber-Foraging for MAR
11.2.4 Network Protocols
Audio and Video Protocols
D2D Multimedia Protocols
Improving General Performance
11.2.5 Discussions
11.3 Network Access
11.3.1 Wireless Networks
HSPA+ (High Speed Packet Access)
LTE (Long-Term Evolution)
LTE Direct
Wi-Fi
Wi-Fi Direct and Wi-Fi Ad Hoc
11.3.2 Future Wireless Architectures and 5G: Promises and Challenges
11.3.3 Upload to Download Ratio on Asymmetric Links: A Delicate Balance
History and Future of Access Networks
Are Symmetric Links Really Necessary?
11.4 Infrastructure and Transport Protocol for MAR
11.4.1 Classful Traffic
11.4.2 Congestion Control, Fairness, and Graceful Degradation
11.4.3 Enforcing Low Latency with Loss Recovery
11.4.4 Multipath
11.4.5 Multi-Server and Distributed Computations
11.4.6 Security and Privacy
11.4.7 Implementation Notes
11.5 Improving Latency at the Application Layer
11.5.1 Mobile AR Pipeline
11.5.2 Commercial SDK Cloud Offloading Procedures
11.5.3 Commercial SDK End-To-End Latency Analysis
11.5.4 Discussion
11.6 Conclusion
11.6.1 Access Link
11.6.2 Network and Transport Layer
11.6.3 Application Layer
11.6.4 Future Challenges
References
Part IV Applications in Arts, Education, and Culture
12 Augmented Reality in Arts Education
12.1 Basic Concepts
12.1.1 Augmented Reality and Arts Education: What Intersection Point?
12.1.2 Arts Education Strategies in the Twenty-First Century
12.1.3 Augmented Reality in Art Teaching at a European Level: What Developments?
12.2 Augmented Reality for Meeting and Making Art
12.2.1 Augmented Reality for Knowledge of Arts
Augmented Reality-Based Art Learning in Museums
Augmented Reality-Based Art Learning in Archaeological and Art Sites
12.2.2 Augmented Reality-Based Learning and Creative Expression
12.3 Digital Environments and Augmented Reality
12.3.1 Augmented Reality as Third Space
12.3.2 Augmented Reality for Creating Digital Artefacts
12.4 The Language of Images and Augmented Reality
12.4.1 The Audio-Visual Language
The Language of the Texts
The Language of Filming
The Language of Lighting
The Language of the Setting
The Language of Characterization
12.4.2 Body Language
The Language of Editing
The Language of Sound
The Language of Music
The Language of Graphics
12.4.3 Educational Experiences
Images and Learning
Digital Images and Media Education
12.4.4 Augmented Reality and Educational Experiences with Arts
12.4.5 Augmented Reality: Art for the Sake of Art
12.5 Digital Competences and Augmented Reality
12.5.1 Augmented Reality and Digital Innovation
12.5.2 Augmented Reality and Digital Competences of Museum Educators
References
13 Augmented Reality's Application in Education and Training
13.1 Development of Augmented Reality
13.2 Defining Augmented Reality
13.3 Pedagogical Framing of Augmented Reality
13.3.1 Location
13.3.2 Task
13.3.3 Role
13.4 Limitations
13.4.1 Technical Issues
13.4.2 Location Detection
13.4.3 Usability
13.4.4 Pedagogy
13.4.5 Health and Safety
13.5 Future Directions for Educational Use of Augmented Reality
References
14 Augmented Reality in Sports and Physical Education
14.1 Introduction
14.2 Status Quo of PE
14.3 Application of IT in PE
14.4 Design of an AR-Assisted Learning System for PE
14.5 Research Methods and Verification
14.6 Learning Outcomes of AR-Assisted Learning
14.7 Motor Skill Acquisition by Using AR-PEclass
14.8 Learning Motivation Stimulated by AR-PEclass
14.9 Conclusion
References
15 Potentiating Learning Through Augmented Reality and Serious Games
15.1 Introduction
15.2 Augmented Reality in Pedagogical Contexts
15.3 Serious Games in Pedagogical Contexts
15.3.1 Educational Learning Approaches
15.3.2 Design of Serious Games for Education
15.3.3 Types of Serious Games Models
15.3.4 Serious Games and User Experiences
15.4 Serious Games and Augmented Reality
15.5 Case Studies
15.5.1 Visualizing Platonic Solids in Augmented Reality
Motivation
Technology and Specifications
Pilot Study
Observations
Discussion
Remarks
15.5.2 CodeCubes
Motivation
Technical Development
Pilot Study
Procedure
Discussion and Results
Study Limitations
Remarks
15.5.3 FootMath
Motivation
Technical Development
Exploring FootMath
Pilot Study
Procedure
Discussion and Results
Remarks
15.6 Closing Remarks
15.7 Conclusions
References
16 Augmented Reality for Cultural Heritage
16.1 Virtual Technologies for Cultural Heritage Legibility
16.2 State of the Art on AR in CH Contexts
16.2.1 Historical Background
16.2.2 Usage of AR Between Real and Virtual
16.3 Relevant Case Studies in CH: Different Approaches to AR
16.3.1 A Selection of Case Studies
Olympia, 2000
Drawings on Glass at Carnuntum (Austria), 2011
Jumieges 3D, Normandy (France), 2013
ViaggiArte: Fontanelle Cemetery, Naples (Italy), 2019
CEMEC: The Box of Stories, EU Project, 2015–2019
The Santa Maria Antiqua Videomapping, Rome (Italy), 2015
The Revealing Flashlight, EU Project, 2013
iMARECULTURE, EU Project, 2016–2020
ARETE, EU Project, 2019–2023
16.3.2 Analytical-Comparative Summary of Use Cases
16.4 Technical Limitations and Challenges of AR for CH
16.5 Multisensorial AR Experiences
16.6 Efficacy and Effectiveness of AR
16.6.1 The Importance of User Experience Design for AR Systems
16.6.2 Efficacy and Effectiveness of AR Systems in Relation to the Context of Use
16.7 Directions and Future Perspectives
References
17 Augmented Reality in Holocaust Museums and Memorials
17.1 Introduction
17.2 Traditional Education of the Holocaust
17.2.1 Ethical Considerations for Education of the Holocaust
17.3 Multimedia Content for Holocaust Education
17.4 Augmented Reality for Holocaust Education
17.5 Ethics of Developing Augmented Reality Apps for Holocaust Museums and Memorials
17.6 Visualization Methods for Digital Reconstruction
17.7 Potential Applications of Augmented Reality for Holocaust Education
17.8 Conclusion
References
18 Augmented Reality into Live Theatrical Performance
18.1 Background
18.2 Augmented Reality in Theatrical Performance
18.2.1 ALICE Project
Technical Components
Technological Affordances in Performance
Safety Considerations
Sensor Considerations
Data Considerations
Considerations for Working with Performers
18.2.2 What the Moon Saw
Agentic Affordances of AR in Theater for Young Audiences
Storytelling Affordances of AR
Considerations for Public Interaction
Multiple Forms of Public Engagement
Actors, Audience, and AR
18.3 Discussion
18.3.1 Potentials for Nonlinear Storytelling
18.3.2 Interfacing for Immersion
18.3.3 Accessibility
18.3.4 Future Directions
18.4 Conclusion
References
19 How do Tourists Evaluate Augmented Reality Services? Segmentation, Awareness, Devices and Marketing Use Cases
19.1 Introduction
19.2 Holistic Augmented Reality Marketing in Tourism
19.2.1 Augmented Reality Marketing
19.2.2 Augmented Reality Marketing in Tourism: Use Cases
AR in the Pre-Booking and Information-Gathering Stage
On-Site AR Experiences
19.2.3 Augmented Reality Marketing in Tourism: Prior Research
19.3 The Role of Augmented Reality in Tourism: Research Questions
19.4 Study
19.4.1 Research Design and Methodology
19.5 Results
19.6 Discussion
19.6.1 General Conclusion
Be Aware that Consumers are Not Aware
AR is Just Getting Started
Focus on Individual Tourist Needs
References
Part V Applications in Engineering and Science
20 Augmented Reality Uses and Applications in Aerospace and Aviation
20.1 History of Augmented Reality in Aerospace
20.2 Applications in Navigation and Guidance
20.2.1 Uses of AR in Manned Navigation and Guidance
20.2.2 Unmanned Navigation and Guidance
20.2.3 Air Traffic Management
20.3 Applications in Engineering, Manufacturing, and Maintenance
20.3.1 Engineering Design and Visualization
20.3.2 Manufacturing and Maintenance Support
Remote and Tele-maintenance
20.4 Applications in Space Operations
20.5 Future Application Concepts
20.5.1 Airport Security
20.5.2 Crew Task Support
20.5.3 In-Flight Entertainment and Communication
20.5.4 Augmented Reality and Artificial Intelligence
20.5.5 Augmented Reality and Haptic Feedback
20.5.6 Augmented Reality Applications for COVID-19 Support
20.6 Concluding Remarks
References
21 Augmented Reality for Building Maintenance and Operation
21.1 The Building Maintenance Instruction Manual
21.2 Augmented Reality and Facility Management
21.3 Systematic Literature Review
21.3.1 Existing Studies
21.3.2 Artifacts Categorization
21.4 Methodology
21.4.1 Identification of the Problem
21.4.2 Design and Development
21.4.3 Artifact Evaluation
21.4.4 Explicitness of Learning
21.4.5 Generalization to a Class of Problems
21.5 Design and Development
21.5.1 Artifact Design
21.5.2 Artifact Development
21.5.3 Artifact Publication
21.6 Evaluation and Findings
21.6.1 General Workload Measurement: NASA TLX
Total Sample Characterization
Workload and Factor Analysis
21.6.2 Workload Analysis Considering Perception Filters
21.7 Generalization
21.8 Conclusion
21.9 Data Availability
References
22 An Augmented Reality Platform for Interactive Finite Element Analysis
22.1 Introduction
22.1.1 Brief Overview of FEA
22.1.2 Augmented Reality
22.1.3 Research Motivations and Objectives
22.2 Research Background
22.2.1 Interactive FEA in VR
22.2.2 Numerical Simulation and Scientific Visualization in AR
22.2.3 Real-Time Finite Element Modeling
22.2.4 Discussion
22.2.5 Summary
22.3 FEA-AR Integrated System
22.3.1 System Design Considerations
22.3.2 System Architecture
22.3.3 AR Environment Setup
Hardware Configuration
Interaction Tools
Coordinate Systems and Transformation
Object Selection Techniques
22.3.4 Summary
22.4 Visualization and Exploration of FEA Results
22.4.1 Scientific Visualization with VTK
22.4.2 Scientific Visualization in AR1pt
Integration of VTK and AR
Occlusion Handling
22.4.3 Data Manipulation and Exploration
22.4.4 Summary
22.5 Real-Time FEA in AR
22.5.1 Real-Time FEA Solution
22.5.2 Computation of Inverse Stiffness Matrix
Matrix Inversion with PCG Method
Matrix Inversion Using External FEA Program
22.5.3 Load Acquisition
Load Management and Conversion
WSN Configuration
Application of Virtual Loads
22.5.4 System Workflow and Time Synchronization
22.5.5 Summary
22.6 Interactive Model Modification
22.6.1 Adding Geometric Models
22.6.2 Local Mesh Refinement
22.6.3 Summary
22.7 System Implementation and Case Studies
22.7.1 Case Study of a Step Ladder
Model Preparation
Data Visualization and Exploration
Real-Time Simulation
Re-Analysis with Model Modification
Local Mesh Refinement
System Response Time
22.7.2 Case Study of a Proving Ring
22.7.3 A Prototype Application for Education
22.7.4 Summary
22.8 Conclusions and Future Readings
22.8.1 Some Open Issues
Enrich the Interactive Model Modification Methods
Apply Model Reduction Techniques for Efficient Analysis
Adapt the System to Mobile AR Platforms
Real-Time Simulation of Soft Objects in AR
Structural Health Monitoring with Finite Element Model Updating
References
23 Augmented Reality in Maintenance: A Review of the State-of-the-Art and Future Challenges
23.1 Augmented Reality Applications in Maintenance
23.2 Summary of Reviews of Augmented Reality in Maintenance
23.3 Literature Review
23.3.1 Applications
23.3.2 Contextual Awareness
23.3.3 Tracking
23.3.4 Human Motion Tracking
23.3.5 User Interface and Interaction
23.3.6 Hardware
23.4 Discussion
23.4.1 Feasibility of AR Application in Industrial Maintenance
23.4.2 Uniform Implementation Method for AR in Maintenance
23.5 Research Gaps Based on Current Technological Developments
23.5.1 A Unified and Robust Design Approach to Achieve Contextual Awareness
23.5.2 UI Design for AR in Remote Maintenance
23.5.3 IoT and Digital Twin Integration
23.5.4 Improvements in User Tracking
23.5.5 Ergonomic Assessment of Current Maintenance Practices
23.5.6 Potential Future Research Focus Areas
Closed-Loop AR-Assisted Maintenance Systems
Context Awareness of AR-Based Maintenance Systems
Automated Authoring of AR-Assisted Maintenance Systems
Ergonomic Consideration and Tracking
Development of More Robust Remote AR Maintenance Systems
IoT and Industry 4.0 Integration
23.6 Conclusion
References
24 Augmented Reality for Maintenance and Repair
24.1 Introduction
24.2 AR for Maintenance
24.3 AR Solutions for Repairs and Troubleshooting
24.3.1 AR and Industry 4.0
24.3.2 Techniques and Methodologies Behind AR-Aided Servicing
24.3.3 Augmentation Strategies for AR-Based Guidance to Repair Procedures
24.3.4 Near-Future Scenarios: Collaborative Repair and Troubleshooting ThroughAR and Virtual Assistants
24.4 Design of a Virtual Toolbox: TheiEngine Framework
24.4.1 Layered Architecture of iEngine
User Layer
Hardware Layer
Low-Level/Interface Layer
Middle-Level/Manager Layer
High-Level/Decision-Making Layer
24.4.2 iEngine Module Architecture
Modules
Services
Plugins
Packages
24.4.3 Novelty of the Developed Approach in iEngine
24.5 Use Case and Experimentation
24.5.1 VTE Modules
24.5.2 VTE Application AI Module
Finite State Machines
Decision Trees
24.5.3 VTE Final Result
24.6 Conclusions
References
25 Augmented Reality for Naval Domains
25.1 Historical Review of Selected Research
25.1.1 Implementations of the Head-Up Display Concept
Aircraft Pilot
Ship Captain
Dismounted Personnel
25.1.2 Maintenance and Repair Applications
Aircraft Manufacturing
Military Maintenance
25.2 Research and Development Efforts
25.2.1 Dismounted Infantry Training
Research at the Naval Research Laboratory
Development via the Office of Naval Research
Evaluation and Testing by the Marine Corps
Discussion of Marine Training Application
25.2.2 Other Applications to Military Operations
GunnAR
Augmented Ship Transits for Improved Decision-Making
Virtual Scientist
Battlespace Visualization
25.2.3 Maintenance and Repair
Service Maintenance Augmented Reality Tools
In-Service Engineering Agent of the Future
Ocean AR
Local Maintenance
Remote Maintenance Aid
25.2.4 Other Applications and Technologies
Vestibular Therapy Using AR
Rapid Prototypes
Human Factors
Color Perception
25.3 Discussion
25.3.1 Augmented Reality in the Navy: A Roadmap
25.3.2 Human Factors Engineering Technical Advisory Group
25.3.3 Lessons Learned
25.4 Conclusions and Future Directions
References
26 Augmented and Mixed Reality for Shipbuilding
26.1 Introduction
26.2 State of the Art
26.2.1 Shipbuilding Welding
26.2.2 Shipbuilding Painting
26.2.3 Shared Information
26.2.4 Step-by-Step Guidance
26.2.5 Design and Construction Assistance
26.2.6 Commercial Developments
26.3 Potential Shipbuilding Tasks to Be Enhanced with AR/MR
26.3.1 Quality Control
26.3.2 Guided Manufacturing
26.3.3 Product and Tool Tracking
26.3.4 Warehouse Management
26.3.5 Predictive Maintenance
26.3.6 Augmented and Mixed Reality Communications
26.3.7 Hidden Area Visualization
26.3.8 Monitoring and Interaction with IIoT Devices
26.3.9 Easy Interaction with Advanced Industrial Software
26.3.10 Structure Visualization
26.3.11 Training
26.3.12 Product and Tool Maintenance
26.4 AR/MR Architectures for Shipbuilding Applications
26.4.1 Traditional AR/MR Communications Architectures
26.4.2 Advanced Communications Architectures
26.5 AR/MR Hardware and Software for Shipbuilding Applications
26.5.1 Ideal AR/MR Device Characteristics
26.5.2 AR/MR Devices for Shipbuilding Applications
26.5.3 Software AR/MR Frameworks for Shipbuilding
26.5.4 Lessons Learned from the Shipbuilding State of the Art
26.6 Main Challenges for the Development of AR/MR Shipbuilding Applications
26.7 Conclusions
References
27 Augmented Reality for Remote Assistance (ARRA)
27.1 Introduction
27.2 Background
27.3 ARRA
27.3.1 ARRA: A Practical Example
27.3.2 Technical Development
27.4 Test Design and Methodology
27.4.1 Validation Case Study
27.4.2 Quantitative Validation Test Methodology
27.5 Analysis and Results
27.6 Discussion
27.7 Conclusion and Future Work
References
Part VI Applications in Health Science
28 Augmented Reality for Computer-Guided Interventions
28.1 From Medical Image Computing to Computer-Assisted Interventions
28.2 Imaging and Anatomical Modeling
28.3 Surgical Vision
28.4 Registration and Pose Estimation
28.5 Physics-Based Modeling
28.5.1 Soft Tissue Biomechanics
28.5.2 Strategies for Real-Time Computation
28.5.3 Beyond Surgery
28.6 Visualization and Perception
28.7 Related Topics
28.7.1 Training and Knowledge Transfer
28.7.2 Surgical Planning
28.8 The Future of AR in Medicine
References
29 Youth and Augmented Reality
29.1 Introduction
29.2 Augmented Reality in Video Games
29.2.1 Brief History of Augmented Reality Video Games (ARGs)
29.2.2 Modern ARGs Utilize Smartphone Technology
29.2.3 Benefits of Augmented Reality Video Games
29.2.4 Downsides to Augmented Reality Games' Growing Popularity
29.2.5 Safety Precautions When Using Augmented Reality Games
29.2.6 The Future of Augmented Reality Games
29.3 Augmented Reality and Social Media
29.3.1 Snapchat
29.3.2 Facebook, Messenger, and Instagram
29.3.3 TikTok
29.3.4 Up and Coming AR: Pinterest
29.3.5 Precautions and Risks of AR Use on Social Media Among Children and Adolescents
29.3.6 The Future of AR on Social Media
29.4 Augmented Reality in Sports
29.4.1 Enhancing Athletic Training
29.4.2 Spectator Interactions
29.4.3 Limitations of Augmented Reality in Sports
29.4.4 The Future of Augmented Reality in Sports
29.5 Augmented Reality in Art
29.5.1 Art Appreciation and Experience
29.5.2 Augmented Reality, Art, and Young Children
29.5.3 Limitations of Art and AR
29.5.4 Future of Art and AR
29.6 Augmented Reality in Education
29.6.1 Preschool
29.6.2 Elementary School
29.6.3 Middle School
29.6.4 High School
29.6.5 College
29.6.6 Graduate Education and Vocational Training
Medical School Students
Physicians and Clinicians
Patients
Other Vocations
29.6.7 Augmented Reality and the Disabled
29.6.8 Cautions
29.7 Augmented Reality in Medical Practice
29.7.1 Educational Opportunities
29.7.2 Therapeutic Uses
Medical Procedures
Patient Communication, Management, and Experience
Nonprocedural Medical Uses
Mental Health
Health/Wellness Goals
29.8 Augmented Reality Safety Precautions
29.8.1 Safety Prior to AR Use
Indoor Considerations
Outdoor Considerations
Driving
Distracted Driver Prevention AR Applications
29.8.2 Safety Measures During AR Use
Injuries and Risks from Spatial Ware and Body Position
Eye Strain
Deep Vein Thrombosis
Injuries from Repetitive Movements
29.8.3 Mental Health
29.9 Considerations and Causes for Concern
29.9.1 Age Considerations for Augmented Reality
Piaget Theory Age-Related Concerns
AAP Recommendations and AR-Specific Supplementation by Authors
29.9.2 Gender Considerations
29.9.3 Pandemic Considerations
29.9.4 Accessibility for All
29.10 Conclusion
References
30 Augmented Reality-Assisted Healthcare Exercising Systems
30.1 Augmented Reality-Assisted Healthcare Exercising Systems
30.2 Related Works
30.2.1 Upper-Extremity Healthcare Exercising Systems
Conventional Methods
VR-Based Systems
Haptic-Based Systems
AR-Based Systems
30.2.2 Upper-Extremity Movement Tracking Methodologies
Sensor-Based Tracking
Bare-Hand Tracking
30.2.3 Assessment of User's Upper-Extremity Functions Recover
Outcome Measurements
Glove-Based Devices for Motor Functions Assessment
30.2.4 Discussion
30.3 System Overview
30.4 Monitoring Module
30.4.1 Motion Tracking
30.4.2 AR-Based Data Glove
30.4.3 Vibration Wrist Band
30.5 Exercise Module and Scoring Module
30.5.1 Flaccid Stage
ROM Exercise for Finger
ROM Exercise for Shoulder
30.5.2 Synergy Stage
Hole Peg Test
Reach-to-Grasp Exercise
Scoring Module for Synergy Stage
30.5.3 Activities of Daily Living Stage
Opening Doors
Arranging the Bookcase
Drawing Water from Faucets
Scoring Module for the Daily Activities
30.6 Usability Experiment
30.6.1 Method
30.6.2 Participants
30.6.3 Experiment Procedure
30.6.4 Design of Pre-test and Post-test
30.6.5 Results and Discussion
30.7 Conclusion and Future Work
References
31 Augmented Reality for Cognitive Impairments
31.1 Introduction
31.2 Background Research
31.2.1 Devices and Tools for AR
31.2.2 Computing Paradigms for AR
31.3 Current Development
31.3.1 Human Memory System
31.3.2 AR Applications in Research of Human Memory and Cognition
Historical Background
AR-Enhanced Memory Aid Systems
31.3.3 AR Applications in Cognitive Load Research
Cognitive Load
AR in Cognitive Load Research
31.3.4 AR Applications for Restoring Perception
Perception-Related Cognitive Impairments
AR-Enhanced Applications in Research of Perception Impairments
31.3.5 Issues and Challenges in the Application of AR for Cognitive Impairments
31.4 Conclusions
References
Part VII Convergence with Emerging Technologies
32 The Augmented Reality Internet of Things: Opportunities of Embodied Interactions in Transreality
32.1 Introduction
32.2 Background
32.2.1 Augmented, Virtual, and Mixed Reality
32.2.2 Artificial Intelligence and Ubiquitous Computing
Intelligent Virtual Agents
Internet of Things
Smart Connected Environments
32.3 The Augmented Reality Internet of Things in Transreality
32.3.1 Convergence of Augmented Reality with Artificial Intelligence and Ubiquitous Computing
Artificial Intelligence and Ubiquitous Computing
Augmented Reality and Artificial Intelligence
Augmented Reality and Ubiquitous Computing
32.3.2 Transreality and The Augmented Reality Internet of Things
32.4 AR-IoT Framework and Interaction Design
32.4.1 Object-Centric AR-IoT Data Management
32.4.2 Scalable AR-IoT Recognition and Tracking
32.4.3 Context-Based AR-IoT Interactions and Content Interoperability
32.4.4 AR-IoT Framework Evaluation
32.5 Opportunities of Embodied Interactions in AR-IoT Environments
32.5.1 Embodied Interactions in AR-IoT Environments
32.5.2 Embodied AR-IoT Agent Prototypes
32.5.3 Embodied AR Agent Insights
32.5.4 Potential Use Cases
32.6 Conclusions
References
33 Convergence of IoT and Augmented Reality
33.1 Introduction
33.1.1 Mixed Reality: A Parallel to the IoT Ecosystem
33.1.2 Toward XRI: A Growing Trend in Hybrid Mixed Reality IoT Systems
33.1.3 Chapter Overview
33.2 A Multidisciplinary Taxonomy for XRI Systems
33.2.1 XRI Taxonomy: Thematic Literature Review
33.2.2 Relating the XRI Taxonomy to XRI System Design Needs from a Multi-disciplinary Perspective
33.3 Contextual Reality (CoRe) Frameworks for XRI System Design
33.3.1 Designing Proof-of-Concept XRI Systems with CoRe
33.3.2 CoRe Framework Insights in Relation to the XRI Taxonomy Perspectives
33.4 Discussion
33.4.1 Directions for Future Work
33.5 Conclusion
References
34 Digital Twin and Extended Reality: Strategic Approach and Practical Implementation
34.1 Introduction
34.2 DTs Enabling Technologies
34.2.1 Enabling Technologies of DT
Cyber-Physical Systems
Internet of Things
Simulation
Extended Reality
Augmented Reality
34.2.2 Digital Twins
34.3 Research Methodology
34.3.1 Literature Review
Planning the Review
Conducting the Review
Reporting the Review
34.4 Results
34.4.1 Findings from the Literature Review
DTs for Asset Management in Manufacturing
DTs for Process Improvement
DTs for Factories
DTs for People
DTs for Product
DTs Benefits
AR in DT Applications
Lifecycle Perspective
34.4.2 DT Strategic Tool
Structure of the Tool
Technological Level
34.5 Practical Implementation of a Combined DT-ER Solution
34.6 Discussion
34.7 Conclusion
References
35 Digital Twins as Foundation for Augmented Reality Applications in Aerospace
35.1 AR-Based System Modeling in Aerospace
35.2 Background
35.2.1 Digital Twin
35.2.2 Digital Twins in Aerospace
35.2.3 Model-Based System Engineering
35.3 AR Digital Twins in Aerospace Applications
35.4 Challenges in AR Digital Twin Implementations
35.4.1 Overlay Precision
35.4.2 Interaction
35.5 Dedicated AR Applications
35.5.1 From Diagrams to AR in Spacecraft Design
35.5.2 On-Orbit Servicing
ar-Based Guidance and Training
35.5.3 Remote Collaboration and Maintenance
35.6 Closing Remarks
References
Index