Affective computing is a nascent field situated at the intersection of artificial intelligence with social and behavioral science. It studies how human emotions are perceived and expressed, which then informs the design of intelligent agents and systems that can either mimic this behavior to improve their intelligence or incorporate such knowledge to effectively understand and communicate with their human collaborators. Affective computing research has recently seen significant advances and is making a critical transformation from exploratory studies to real-world applications in the emerging research area known as applied affective computing.
This book offers readers an overview of the state-of-the-art and emerging themes in affective computing, including a comprehensive review of the existing approaches to affective computing systems and social signal processing. It provides in-depth case studies of applied affective computing in various domains, such as social robotics and mental well-being. It also addresses ethical concerns related to affective computing and how to prevent misuse of the technology in research and applications. Further, this book identifies future directions for the field and summarizes a set of guidelines for developing next-generation affective computing systems that are effective, safe, and human-centered.
For researchers and practitioners new to affective computing, this book will serve as an introduction to the field to help them in identifying new research topics or developing novel applications. For more experienced researchers and practitioners, the discussions in this book provide guidance for adopting a human-centered design and development approach to advance affective computing.
Author(s): Leimin Tian, Sharon Oviatt, Michal Muszynski, Brent Chamberlain, Jennifer Healey, Akane Sano
Series: ACM Books, 41
Publisher: Association for Computing Machinery
Year: 2022
Language: English
Pages: 308
City: New York
Applied Affective Computing
Contents
List of Figures
List of Tables
Preface
Acknowledgments
1 Introduction to Applied Affective Computing
1.1 Affective Computing
1.2 Applied Affective Computing
1.2.1 Challenges of Applied Affective Computing
1.2.1.1 Developing a Human-centered Affective Computing System
1.2.1.2 Developing a Reliable Affective Computing System
1.2.1.3 Developing an Adaptive Affective Computing System
1.2.1.4 Developing an Integrated Affective Computing System
1.2.1.5 Developing an Ethical Affective Computing System
1.2.2 A Living Ontology of Affective Computing
1.3 Book Overview and Highlights
1.4 Contributions
2 Emotions as Studied in Psychology and Cognitive Science
2.1 Theories of Emotion
2.1.1 The Evolutionary Approach
2.1.2 The Appraisal Approach
2.1.3 The Constructionism Approach
2.2 Emotion as an Adaptive Function
2.3 Emotion as a Social Communicative Function
2.3.1 Empathy
2.3.2 Social Intelligence and Socio-affective Competence
2.3.3 Social Norms
2.4 Summary
3 Machine Learning Approaches for Applied Affective Computing
3.1 Machine Learning for Affective Computing
3.2 Deep Machine Learning for Affective Computing
3.3 Multimodal Representation of Affect: Knowledge-inspired versus Data-driven Features
3.4 Unimodal Features for Affect Recognition
3.4.1 Visual Modality—Facial Expressions and Body Gestures
3.4.1.1 Facial Expression
3.4.1.2 Body Gestures
3.4.2 Deep Learning-based Visual Feature Extraction
3.4.3 Audio Modality
3.4.4 Deep Learning-based Audio Feature Extraction
3.4.5 Physiological Modality
3.4.6 Deep Learning-based Physiological Feature Extraction
3.4.7 Linguistic Modality
3.4.8 Deep Learning-based Linguistic Feature Extraction
3.5 Multimodal Emotion-aware Systems
3.6 Evaluation of Emotion-aware Systems
3.7 Synthesis of Emotion and Emotional Behaviors
3.8 Discussion
3.8.1 Context Matters
3.8.2 Emotion Representation: Categories versus Dimensions
3.8.3 Perceived versus Felt Affect
3.9 Conclusion
4 Multimodal Data Collection and Processing for Applied Affective Computing
4.1 Multimodal Data Collection
4.1.1 Emotion Labels
4.1.2 Multimodal Data
4.1.2.1 Facial Expression
4.1.2.2 Speech
4.1.2.3 Body Movement
4.1.2.4 Text
4.1.2.5 Physiological Signals
4.1.2.6 Context Information
4.2 Multimodal Data Processing
4.2.1 Preprocessing
4.2.2 Feature Extraction
4.2.3 Data Reduction/Selection
4.2.4 Feature Normalization
4.3 Multimodal Data Fusion
4.3.1 Model-agnostic Approaches
4.3.1.1 Feature Level Fusion (Early Fusion)
4.3.1.2 Intermediate Fusion
4.3.1.3 Late Fusion (Decision Level Fusion)
4.3.2 Model-based Approaches
4.3.2.1 Kernel Learning
4.3.2.2 Graphical Models
4.3.2.3 Neural Networks
4.3.2.4 Multimodal Deep Autoencoders
4.4 Conclusion and Future Work
5 Emotion Recognition in the Wild
5.1 Modalities
5.1.1 Emotion Recognition from Audio and Visual Modality
5.1.2 Emotion Recognition from Physiological Signals
5.2 Deep Learning for Emotion Recognition in the Wild
5.3 Emotion Representation in the Wild
5.4 Evaluation of Affect Recognition in the Wild
5.5 The Role of Context in Affect Recognition in the Wild
5.6 The Role of Environment in Affect Recognition in the Wild
5.7 Discussion
5.7.1 Emotion Annotations: Self-report versus Third-person Annotations
5.7.2 Long-term Research on Affect in the Wild
5.7.3 Ethical and Privacy Issues of Affect Recognition in the Wild
5.8 Conclusion
6 Reinforcement Learning and Affective Computing
6.1 Reinforcement Learning
6.1.1 Bandits and the Exploration–Exploitation Tradeoff
6.1.2 State Valuation
6.2 Affective Computing
6.2.1 Theories of Emotion
6.2.2 Behavioral Psychology
6.3 Extending the Reinforcement Learning Framework
6.3.1 Modeling Agent, Environment, and Critic
6.3.2 A Multilayered Affective Critic
6.3.3 Dynamic Goals and Rewards
6.3.4 Prioritization
6.4 Conclusions
7 Synthesizing Natural and Believable Emotional Expressions
7.1 Introduction
7.1.1 Historical Context
7.2 Emotional Expression Models
7.3 Synthesizing Emotional Expressions
7.4 The Role of Agency
7.5 Additional Considerations and Challenges
8 Emotion-aware Human–Robot Interaction and Social Robots
8.1 Developing Emotion-aware Human–Robot Interaction Systems
8.2 Improving Social Intelligence of a Robot by Addressing Social Errors in Human–Robot Interaction
8.3 Case Study: Prompting Human Assistance in Human–Robot Collaboration Using Robots' Emotional Expressions
8.3.1 Emotional Model of the Robot
8.3.2 Human–Multirobot Collaboration Task: The Tower Construction Game
8.3.3 Influence of Artificial Emotions on Human–Multirobot Collaboration
8.3.4 Validity and Expressiveness of the Robots' Emotional Expressions
8.3.5 Influence of Robots' Emotional Expressions on Human's Perception
8.3.6 Summary
8.4 Case Study: Understanding Users' Expectations of Robots in Public Spaces
8.4.1 Robots in Today's Public Spaces
8.4.2 Adopting Participatory Design to Understand Expectations and Perceptions of Robots in Public Spaces
8.4.3 Summary
8.5 Challenges in Affective HRI
8.6 Guidelines for Developing Affective HRI Systems and Social Robots
9 Affective Computing for Enhancing Well-Being
9.1 Motivation
9.2 Sensing and Analytics
9.3 Examples of Well-Being Studies
9.3.1 MIT Friends and Family
9.3.2 MoodScope
9.3.3 Student Life
9.3.4 SNAPSHOT Study
9.3.5 Detecting Momentary Affective Changes in Daily Life
9.4 Sensing Beyond the Phone
9.4.1 Stationary Compute
9.4.2 Home Monitoring
9.4.3 In Vehicle
9.4.4 Cross-Platform Integration
9.5 Actions and Interventions
9.5.1 Conversing
9.5.2 Reminding and Recommending
9.5.3 Behavior Modification
9.5.4 Positive Computing
9.6 Conclusions
10 Applied Affective Computing in Built Environments
10.1 Setting the Foundation Toward Emotionally Aware Planning and Design
10.1.1 Design and Evaluation of Urban Spaces
10.1.2 Emotion and Perception
10.1.3 Sensors and Sensor Fusion
10.2 Case Study: Passive Responses to Urban Infrastructure
10.3 Experimental Task
10.4 Data
10.4.1 Human Subjects
10.4.2 Built Environment
10.5 Approach
10.5.1 Data Normalization
10.5.2 Machine Learning Models
10.6 Results
10.7 Conclusions
10.8 Guidelines for Affective Computing in Built Environments
11 Addressing Ethical Issues of Affective Computing
11.1 Ethical Concerns of Affective Computing
11.2 A Fair System
11.2.1 Biases in Data and Models
11.2.2 Increasing Fairness
11.3 A Privacy-preserving System
11.3.1 Issues of Invasive Surveillance
11.3.2 Privacy Preservation
11.4 A Transparent System
11.4.1 Overtrust Toward Intelligent Systems
11.4.2 Supporting Informed Decisions
11.5 A Beneficial System
11.5.1 Understanding Human Preferences
11.5.2 Contradicting Interests
11.6 A Responsible System
11.6.1 Accountability
11.6.2 Governance
11.7 Conclusion
12 Future of Affective Computing and Applied Affective Computing
12.1 Applied Affective Computing: Guidelines for Best Practice
12.2 Example Uses of the Guidelines
12.2.1 Scenario-based Discussion
12.2.1.1 Example A: Novice Researcher
12.2.1.2 Example B: Experienced Researcher
12.2.1.3 Example C: Practitioner
12.2.2 Examples in Existing Studies
12.3 Open Challenges and Future Directions
12.3.1 Toward Rigorous Affective Computing Research Methodologies
12.3.1.1 Hypotheses Grounded in Emotion and Communication Theories
12.3.1.2 Ecological Validity
12.3.1.3 Evaluation with the Right Metrics
12.3.1.4 Reproducibility and Generalizability
12.3.2 Toward Personalized Affective Computing
12.3.3 Toward Adaptive Affective Computing
12.3.4 Toward Embodied Affective Computing
12.3.5 Application Domains of Interests
12.3.5.1 Healthcare
12.3.5.2 Education
12.3.5.3 Business
12.3.5.4 Design
12.3.5.5 Security
12.3.5.6 Entertainment
12.4 Summary
Bibliography
Authors' Biographies
Index
