This handbook is the first book ever covering the area of Multimodal Learning Analytics (MMLA). The field of MMLA is an emerging domain of Learning Analytics and plays an important role in expanding the Learning Analytics goal of understanding and improving learning in all the different environments where it occurs. The challenge for research and practice in this field is how to develop theories about the analysis of human behaviors during diverse learning processes and to create useful tools that could augment the capabilities of learners and instructors in a way that is ethical and sustainable. Behind this area, the CrossMMLA research community exchanges ideas on how we can analyze evidence from multimodal and multisystem data and how we can extract meaning from this increasingly fluid and complex data coming from different kinds of transformative learning situations and how to best feed back the results of these analyses to achieve positive transformative actions on those learning processes. This handbook also describes how MMLA uses the advances in machine learning and affordable sensor technologies to act as a virtual observer/analyst of learning activities. The book describes how this “virtual nature” allows MMLA to provide new insights into learning processes that happen across multiple contexts between stakeholders, devices and resources. Using such technologies in combination with machine learning, Learning Analytics researchers can now perform text, speech, handwriting, sketches, gesture, affective, or eye-gaze analysis, improve the accuracy of their predictions and learned models and provide automated feedback to enable learner self-reflection. However, with this increased complexity in data, new challenges also arise. Conducting the data gathering, pre-processing, analysis, annotation and sense-making, in a way that is meaningful for learning scientists and other stakeholders (e.g., students or teachers), still pose challenges in this emergent field. This handbook aims to serve as a unique resource for state of the art methods and processes.
Chapter 11 of this book is available open access under a CC BY 4.0 license at link.springer.com.
Author(s): Michail Giannakos, Daniel Spikol, Daniele Di Mitri, Kshitij Sharma, Xavier Ochoa, Rawad Hammad
Publisher: Springer
Year: 2022
Language: English
Pages: 361
City: Cham
Contents
Part I Introduction to MMLA
Introduction to Multimodal Learning Analytics
1 Introduction
2 Background and Brief History: What Have We Learned and Where Do We Go Now?
2.1 MMLA Brief History and Community Development
2.2 MMLA Growth and Activities
2.3 MMLA Challenges
2.3.1 Coalescing of Multimodal Data and Advanced Analysis Methods to Support Learning
2.3.2 Methodological and Practical Breakthroughs Through “Multimodality” to Support Learning
2.3.3 MMLA Capabilities to Support Teaching and Learning
2.3.4 From Experimentation, to Use and Adoption of MMLA
3 Contributions and Themes on MMLA
3.1 MMLA for Design
3.2 MMLA for Feedback and Regulation
3.3 MMLA to Support Theory and Pedagogy
3.4 MMLA Approaches, Architectures and Methodologies
3.5 MMLA and Affective States
3.6 Privacy and Ethics of MMLA
3.7 The Past, Present, and Future of MMLA
4 Conclusions and the Way Ahead
References
In This Book
Part II MMLA for Design
Multimodal Learning Analytics and the Design of Learning Spaces
1 Introduction
2 Research on Learning Spaces: From Traditional to Computer-Based Analysis
3 Examples of MMLA Studies Focused on the Effects of Learning Spaces
3.1 The PELARS Project: Standing for Better Collaboration
3.2 The SmartLET Project: Studying the Interplay of Table Design and Educational Level
3.3 Moodoo: Characterising Teachers' Positioning in the Classroom
4 Discussion
4.1 MMLA Potential for Informing the Design of Learning Spaces in Future Research
4.2 MMLA Limitations to Inform the Design of Learning Spaces
4.3 Broadening Use of MMLA Systems
5 Concluding Remarks
References
Part III MMLA for Feedback and Regulation
Multimodal Systems for Automated Oral Presentation Feedback: A Comparative Analysis
1 Introduction
2 Anatomy of a Multimodal System for Oral Presentation Automated Feedback
3 Comparative Technical Analysis of OPAF Systems
3.1 Existing Systems
3.2 Recording
3.3 Extraction
3.4 Analysis
3.5 Feedback
3.6 Conclusions
4 Evaluation and Deployment
4.1 Evaluation
4.2 Levels of Deployment
5 Conclusions and Next Steps
References
Modeling the Complex Interplay Between Monitoring Events for Regulated Learning with Psychological Networks
1 Introduction
2 Studying Monitoring as a Part of Self-Regulated Learning
3 Physiological Activity in Relation to Monitoring
4 Implementing Network Analysis to Capture the Complex Process of Monitoring and Physiological Arousal
4.1 Recent Research Utilizing Psychological Networks
4.2 Illustrating Network Analysis Methods in Practice with Three Data Examples
5 Discussion and Future Prospects
5.1 Prospects of Multimodal Learning Analytics
5.2 Prospects of Psychological Networks
References
The Role of Metacognition and Self-regulation on Clinical Reasoning: Leveraging Multimodal Learning Analytics to Transform Medical Education
1 Introduction
1.1 Assessing and Evaluating Competency in Clinical Reasoning
1.2 Multimodal Learning Analytics in Medical Education
2 Socio-Cognitive Cyclic Model of Self-regulated Learning
2.1 Forethought Phase
2.2 Performance Phase
2.3 Self-reflection Phase
3 Implications of Multimodal Learning Analytics to Improve Medical Education
3.1 Detecting Metacognition and Self-regulated Learning Processes
3.2 Modeling Metacognition and Self-regulated Learning Processes
3.3 Tracing Metacognition and Self-regulated Learning Processes
3.4 Fostering Metacognition and Self-regulated Learning Processes
References
Part IV MMLA to Support Theory and Pedagogy
Intermodality in Multimodal Learning Analytics for Cognitive Theory Development: A Case from Embodied Design for Mathematics Learning
1 Introduction
1.1 Overview of the MIT-P Project
1.2 Theoretical Framework: Intermodal Perception
2 Multimodal MIT-P Analyses: A Brief History
2.1 Hand Movements
2.2 Eye Movements
2.3 RQA Analysis
3 From Multimodal Gaze and Hand Movement to the Intermodal Emergence and Stabilization of Attentional Anchors: An RQA Case Study
3.1 Research Question
3.2 Methods
3.3 Results
3.3.1 RQA Analysis
4 Discussion
4.1 Interpretation of Findings
4.2 Theoretical Implications
4.3 Methodological Implications
4.4 Practical Implications
4.5 Limitations
4.6 Future Directions
5 Conclusion
References
Bridging the Gap Between Informal Learning Pedagogy and Multimodal Learning Analytics
1 Background
2 Pedagogy of Informal Learning
2.1 Context
2.2 Learning Theories and Pedagogical Approaches
2.2.1 Behaviorism
2.2.2 Cognitivism
2.2.3 Constructivism
2.3 Reflections
3 Multimodal Learning Analytics
3.1 Context
3.2 Informal Learning Modalities
3.3 Underpinning Technicalities
3.4 Multimodal Learning Analytic Challenges
4 Discussion
5 Conclusion
References
Part V MMLA Approaches, Architectures and Methodologies
Multimodal Learning Experience for Deliberate Practice
1 Introduction
2 Multimodal Learning Theories
2.1 Embodied Learning
2.2 Deliberate Practice
3 Engineering Aspect
3.1 Architecture Overview
3.2 Interaction Layer
3.3 Data Layer
3.4 Feedback Layer
3.5 Task Layer
3.6 The MLX System into Teaching
4 Research Methodologies
4.1 First Iteration
4.2 Second Iteration
4.3 Following Iterations
5 Application Use Cases
5.1 Presentation Trainer
5.2 CPR Tutor
5.3 Calligraphy Tutor
5.4 Table Tennis Tutor
5.5 Astronaut Training
5.6 Commonalities and Differences
6 Conclusions and Future Work
References
CDM4MMLA: Contextualized Data Model for MultiModal Learning Analytics
1 Introduction
2 Developing MMLA Solutions in Authentic Settings: Lifecycle and Challenges
3 Review of Related Contextualized Data Models
4 Contextualized Data Model for MultiModal Learning Analytics (CDM4MMLA)
4.1 Information Model of CDM4MMLA
4.2 Description of the CDM4MMLA
5 Applying CDM4MMLA to Authentic MMLA Scenarios
5.1 Case 1: MUlti-Modal Teaching and Learning Analytics (MUTLA) Dataset Scenario
5.2 Case 2: The MULTISIMO Corpus Scenario
5.3 Case 3: MMLA for a Secondary School English Course Scenario
6 Discussion
7 Conclusions and Future Work
References
A Physiology-Aware Learning Analytics Framework
1 Introduction
2 State of the Art
3 A Framework for Physiology-Aware Learning Analytics
3.1 A Stress-Sensitive Pedagogical Agent in PHYLA
3.2 PlugIn for HRV Data
3.3 Physiological Backend: Physiological Parameters of Stress
3.4 A Dialogue Concept for a Stress-Sensitive Pedagogical Agent
4 Evaluation
4.1 Design
4.2 Results
4.2.1 Study Participants
4.2.2 Hypothesis 1
4.2.3 Hypothesis 2
4.2.4 Hypothesis 3
5 Conclusion, Discussion, and Future Work
Appendix
References
Part VI MMLA and Affective States
Once More with Feeling: Emotions in Multimodal Learning Analytics
1 Introduction
2 What Are Emotions?
3 Emotions and Learning
3.1 Emotions and Learning Outcomes
3.1.1 Emotions and Learning Processes
4 Measuring Emotions
5 An Applied Example
5.1 Data Sources
5.2 Analyses and Workflow
5.2.1 Statistical Analyses
5.3 Results
6 Discussion
6.1 Valence Based Results
6.2 Discrete Emotions-Based Results
6.3 Challenges in Multi-Modal Affect Detection
7 Conclusion
References
Part VII Privacy and Ethics of MMLA
The Evidence of Impact and Ethical Considerations of Multimodal Learning Analytics: A Systematic Literature Review
1 Introduction
2 Background and Related Work
2.1 Ethical Considerations of Learning Analytics
3 Methodology
4 Results
4.1 Data Modalities Used in MMLA Research
4.2 Existing Evidence on the Use of MMLA to Support Educational Outcomes
4.3 Ethical Considerations Highlighted and Addressed in MMLA Research
5 Discussion
5.1 Limitations
6 Conclusions
Appendix A
WOS
Scopus
ACM
IEEE
Appendix B
Appendix C
Appendix D
Appendix E
References
Part VIII The Past, Present, and Future of MMLA
Sensor-Based Analytics in Education: Lessons Learned from Research in Multimodal Learning Analytics
1 Introduction
2 Sensor-Based Analytics in Education
2.1 SBA Qualities
2.2 SBA Objectives
2.3 SBA Challenges (Barriers of Adoption)
2.4 SBA Opportunities
3 The Role of SBA in Different MMLA Case Studies
3.1 Case Study 1: SBA to Improve Accuracy of Learner Models
3.2 Case Study 2: SBA to Capture Information Unobtrusively
3.3 SBA and Embodied Learning
4 Discussion: Towards SBA Adoption in Education
4.1 Future Research on Methodological and Practical Aspects
4.2 Future Research on Ethical Aspects
Appendix: Categorization of Challenges and Opportunities, Across the Three Case Studies
References
Framing the Future of Multimodal Learning Analytics
1 Introduction
2 MMLA Research Paradigms
2.1 MMLA for Theorizing
2.2 MMLA for Practice
2.3 MMLA for Interactivity
2.4 Paradigms Summary
3 Realizing Ethical Practices Across Different Aspects of an MMLA Research Project
3.1 Data Collection: Multimodal Data Control/Data Ownership
3.2 Data Analysis: Limitations in Prediction from Multimodal Data/Commitment to Fair and Ethical Language When Talking About Research Participants
3.3 Data Dissemination: Transparency and Benefit/Moving Away from Research as an Extractive Process
3.4 Commitments Summary
4 Re-conceptualizing Learning Through an MMLA Perspective
4.1 Methods for Data Analysis with Increased Data Privacy and Control
4.2 Developing New Standards for Non-traditional Metrics
4.3 Thinking About These Standards over Different Time Scales, Levels of Granularity, and Contexts
4.4 Moving Beyond Randomized Control Trials as the Gold Standard
4.5 Embracing Deep, Nuanced, and Potentially Divergent Pictures of the Learner
5 Conclusion
References
Index
