Methods for Researching Professional Learning and Development: Challenges, Applications and Empirical Illustrations

Series Editors´ Foreword
Contents
Chapter 1: Addressing Methodological Challenges in Research on Professional Learning and Development
1.1 Professional Learning and Development as a Research Field
1.2 Challenges Connected to PLD Research
1.3 Methodological Advancements: Alternative Approaches to Challenges
1.4 Structure of the Book
References
Part I: Methods for Data Collection
Chapter 2: Applying the Experience Sampling Method to Research on Workplace Learning
2.1 Introduction
2.2 The Experience Sampling Method (ESM) - A Brief Characterisation
2.2.1 Conceptual Elements of ESM
2.2.2 Level 1 and Level 2 Variables (Within-Person and Between-Person Processes)
2.2.3 Study Designs and Schedules for Data Collection
2.2.4 Data Analysis
2.2.5 Potentials and Limitations of ESM
2.3 Examples for the Use of ESM to Research Workplace Learning
2.3.1 Example 1: Workplace Learning (Daniels et al., 2009, 2011)
2.3.2 Example 2: Analysing Knowledge Sharing Behaviour in Business Centres (Weijs-Perrée et al., 2020)
2.3.3 Example 3: Analysing Motivational Processes in Vocational School Classrooms (Seifried & Klüber, 2006)
2.3.4 Example 4: Measuring Non-cognitive Facets of Competence in Assessment Situations (Rausch et al., 2016, 2019; Seifried et...
2.4 Conclusion and Outlook
References
Chapter 3: Uncovering Informal Workplace Learning by Using Diaries
3.1 Introduction
3.2 Diaries as a Research Instrument
3.2.1 Validity of Diary Data on Informal Workplace Learning
3.2.2 Types of Research Questions
3.2.3 Parameters of the Research Diaries
3.2.3.1 Item Format, Adaptability, and Recording Device
3.2.3.2 Sampling Method
3.2.3.3 Reference Time and Time Delays
3.2.3.4 Study Period and Overall Research Design
3.2.4 Reporting Diary Research
3.3 Illustrating Diary Research
3.3.1 Study I: Using a Paper-Based Diary to Investigate Emotions During Error Situations
3.3.1.1 Context and Research Questions
3.3.1.2 Method
3.3.1.3 Findings
3.3.1.4 Conclusions from Study I
3.3.2 Study II: Using Voice-Recorded Diaries to Research Nurses´ Learning at Student-Run Hospital Wards
3.3.2.1 Context and Research Questions
3.3.2.2 Method
3.3.2.3 Findings
3.3.2.4 Conclusions from Study II
3.4 Conclusion
References
Chapter 4: Using the Vignette Technique to Increase Insight into Professional Development at Work
4.1 Introduction
4.2 The Vignette Technique
4.2.1 Characteristics of Vignettes
4.2.2 Instructions and Answering Modes
4.3 Use of the Vignette Technique
4.3.1 Example 1: Learning from Errors
4.3.2 Example 2: Team Mental Models
4.3.3 Example 3: Complexity of Work Tasks and Informal Learning Activities
4.3.4 Example 4: Uncertainty and Proactive Work Behaviour
4.4 Development of Vignettes
4.5 Discussion and Conclusions
References
Chapter 5: Integrating Self-Reports and Electrodermal Activity (EDA) Measurement in Studying Emotions in Professional Learning
5.1 Introduction
5.2 Emotions Related to Learning
5.3 EDA Measurements in Emotion and Learning Research
5.4 Integrating Self-Reports and EDA: An Empirical Study on Emotions in Professional Learning
5.4.1 Research Design and Data Collection
5.4.2 Data Analysis
5.4.3 Integrating EDA and Self-Reports from EC and SRI
5.5 Discussion
5.5.1 Future Steps
References
Chapter 6: Multimodal Stress Assessment in Working and Learning Contexts Using Physiological, Observational and Experience-Bas...
6.1 Topic and Structure of the Chapter
6.2 Stress-Inducing and Stress-Buffering Potentials of Working and Learning in the Workplace
6.3 Physiological Stress Responses
6.3.1 Hypothalamic-Pituitary-Adrenal Axis and Cortisol Response to Workplace Stress
6.3.2 Autonomic Nervous System and Cardiovascular Stress Markers
6.4 Example Studies
6.4.1 Example Study 1: Effects of Work-Related Problem Solving on Cortisol Response and Cardiovascular Stress Markers
6.4.1.1 Sample and Method
6.4.1.2 Measures
6.4.1.3 Findings
6.4.2 Example Study 2: Time-Lagged Effects of Teachers´ Experienced Classroom Demands on Autonomic Stress Reactions
6.4.2.1 Sample and Method
6.4.2.2 Measures
6.4.2.3 Findings
6.4.3 Example Study 3: Interactions Between Baseline Cortisol Levels of Trainee Industrial Clerks and Classroom Demands Concer...
6.4.3.1 Sample and Method
6.4.3.2 Measures
6.4.3.3 Findings
6.5 Framework for Multimodal Assessment and Evaluation of Learning and Working Environments
6.6 Concluding Recommendations for Research Practice
References
Chapter 7: Combining Physiological and Experiential Measures to Study the Adult Learning Experience
7.1 Introduction
7.2 Adult Learning Research: From Traditional to Multimodal Combinations of Methods in Researching Learning Experience
7.2.1 Combining Methods to Explore Learning Experience
7.2.2 Multimodal Research of the Adult Learning Experience
7.2.2.1 Multimodal Studies on Learning Experience and Emotions
7.2.2.2 Multimodal Studies on Learning Experience and Cognitive Processes
7.2.2.3 The Role of Multimodality in Studying SBL Experience
7.2.3 Building a Theoretical Framework for the Multimodal Study of Learning
7.3 Case: Multimodal Study of the Forestry SBL Experience
7.3.1 Context and Participants
7.3.2 Research Data Collection
7.3.2.1 Protocol
7.3.2.2 Data Collection Methods
7.3.3 Data Analysis
7.3.3.1 Analysis of Interview Data
7.3.3.2 Preparing Interview Data for Multimodal Integration
7.3.3.3 Analysis of Physiological and Neurophysiological Data
7.3.3.4 Approaching Integration Across Data Modalities
7.4 Discussion
7.4.1 Understanding Learning Experience Requires Crossing Disciplinary Boundaries
7.4.2 Combining Traditional Approaches with Physiological and Neurophysiological Recordings
7.4.3 A New Theoretical Framework Requires Building Consistent Empirical Evidence of Learning Experiences in Natural Contexts
References
Chapter 8: Measuring Professional Competence Using Computer-Generated Log Data
8.1 Introduction
8.2 Investigating Differences in Problem-Solving Processes Using Log Data
8.2.1 Theory-Based Approaches: Identification of A Priori Defined Problem-Solving Strategies
8.2.2 Data-Driven Approaches: Exploration of Unknown Data Patterns
8.2.3 Professional Problem Solving: Analysing Log Data from a Digital Office Simulation
8.2.4 Identification of Efficiency Barriers During the Problem-Solving Process
8.3 Measuring Diagnostic Problem-Solving Strategies of Car Mechatronics Apprentices
8.3.1 Introduction: Why Do We Assess Diagnostic Problem-Solving Strategies in the Domain of Car Mechatronics?
8.3.2 Methods: How Can We Assess Diagnostic Problem-Solving Strategies in the Domain of Car Mechatronics?
8.3.2.1 A Priori Definition of Critical Problem-Solving Behaviour
8.3.2.2 Deriving Behavioural Patterns for each Strategy
8.3.3 Log Data Analysis and Selected Results
8.3.4 Discussion
8.4 Lessons Learned
8.4.1 Assessment and Pre-processing of Log Data
8.4.2 Analysis of Log Data
8.4.3 Interpretation of Log Data
8.5 Outlook
References
Chapter 9: Investigating Interaction Dynamics: A Temporal Approach to Team Learning
9.1 Introduction
9.2 Team Learning and Interaction Dynamics
9.2.1 What Are Interaction Dynamics?
9.2.2 Connecting Team Learning Processes to Interaction Dynamics
9.3 Researching Interaction Dynamics: How Do I Gather Data?
9.3.1 Data Gathering Considerations
9.3.2 Data Coding Considerations
9.3.3 Towards a Combined Data Gathering and Coding Methodology
9.3.3.1 Behaviour Sensor Systems
9.3.3.2 Supervised and Unsupervised Machine Learning
9.4 Researching Interaction Dynamics: How Do I Analyse My Data?
9.4.1 Developing a Framework for Choosing a Data Analysis Technique
9.4.2 Pool of Temporal Analysis Techniques
9.5 A Temporal Approach to Team Learning: Closing Remarks
9.5.1 Key Takeaways
References
Part II: Methods for Data Analysis
Chapter 10: Bayesian Statistics in the Research Field of Professional Learning and Development
10.1 Introduction
10.2 Introduction to Bayesian Statistics
10.2.1 Bayes´ Rule
10.2.1.1 An Example of Applying Bayes´ Rule
10.2.2 Comparison of Frequentist and Bayesian Correlation Analysis
10.3 Bayesian Multilevel Path Analysis
10.3.1 Participants and Measures
10.3.2 Analysis Approach
10.3.3 Analysis Settings
10.3.4 Statistical Procedures
10.3.5 Data Aggregation
10.3.6 Descriptive Statistics
10.3.7 Results
10.3.7.1 Random Intercept Model
10.3.7.2 Random Slope Model
10.3.8 Conclusions
10.4 Discussion
References
Chapter 11: A Primer to Latent Profile and Latent Class Analysis
11.1 Introduction
11.2 Conceptual Overview
11.2.1 Family Relations
11.2.2 A Person-Centred Conceptual Perspective
11.2.3 The LPA and LCA Models
11.2.4 Steps in LPA/LCA
11.3 Implementation of LPA and LCA
11.3.1 Example 1: Univariate LPA of Student Achievement
11.3.2 Example 2: Multivariate LPA of Work Recovery
11.3.3 Example 3: LCA of Teachers´ Participation in Professional Development
11.4 Conclusions
11.4.1 Some Words of Caution
11.4.2 Notes on Software and Further Reading
References
Chapter 12: PLS-Based Structural Equation Modelling: An Alternative Approach to Estimating Complex Relationships Between Unobs...
12.1 Introduction
12.2 Structural Equation Modelling in a Nutshell
12.2.1 Relationships Between Constructs and Indicators: The Measurement Model
12.2.2 Relationship Types Specified Between Constructs: The Structural Model
12.2.3 Estimating Structural Equation Models
12.3 Partial Least Square SEM
12.3.1 PLS Algorithm
12.3.2 Practical Issues Using PLS-SEM
12.4 Illustrating PLS-SEM with a Replication Study
12.4.1 Motivation and Theoretical Framework
12.4.2 Method
12.4.3 Discussion
12.5 Conclusion
References
Chapter 13: Participant´s Video Annotations as a Database to Measure Professional Development
13.1 Introduction
13.2 Video Annotation
13.3 Video Annotation as a Data Collecting Method
13.3.1 Comparing Video Annotation with Other Methods of Data Collection
13.3.2 Video Annotation from the Perspective of Quality Criteria
13.4 Practical Application of Video Annotation
13.4.1 Planning
13.4.2 Data Collection
13.4.3 Analysis
13.4.4 Practical Possibilities
13.5 Example Study
13.5.1 Framework
13.5.2 Data Gathering
13.5.3 Data Analysis
13.5.4 Results
13.6 Conclusions
References
Chapter 14: Data Mining and Analytics in the Context of Workplace Learning: Benefits and Affordances
14.1 Introduction
14.2 Data Mining and Analytics in the Context of Education
14.2.1 Educational Data Mining
14.2.2 Learning Analytics
14.3 Benefits and Affordances
14.3.1 Benefits for Workplace Learning
14.3.2 Affordances and Challenges
14.4 Conclusion
References
Chapter 15: Addressing `Wicked Problems´ Using Visual Analysis
15.1 Introduction
15.2 Theoretical Background
15.2.1 Theory of Wicked Problems
15.2.2 A Wicked Problem Perspective on Professional Learning and Development
15.2.3 Tackling Wicked Problems with Visual Analysis
15.2.4 What Is Not Visual Analysis?
15.3 Illustration
15.3.1 Evaluating Missingness in Longitudinal Multilevel Data on Teachers´ Engagement
15.3.2 Missingness in Longitudinal Data: A Wicked Problem
15.3.3 The Evolution of Teachers´ (Team) Engagement
15.3.3.1 Background
15.3.3.2 Goals and Research Questions
15.3.3.3 Sample
15.3.3.4 Instruments
15.3.4 Visualising Longitudinal Multilevel Missingness
15.3.4.1 Goals of the Visualisation
15.3.4.2 Design Process
15.3.4.3 The Visualisation
15.3.5 Findings
15.4 Discussion
15.4.1 Contributions and Pitfalls of a Visual Analytical Approach
15.4.2 Future Perspectives for Visual Analytics in Professional Learning and Development
References
Part III: Research Approaches
Chapter 16: Delphi-Technique as a Method for Research on Professional Learning
16.1 Introduction
16.2 Characteristics
16.2.1 Standard Procedure
16.2.2 Group of Participants
16.2.3 Iterative Procedure and Feedback
16.2.4 Anonymity Between the Participants
16.3 Challenges, Advantages, Disadvantages
16.3.1 Challenges
16.3.2 Advantages
16.3.3 Disadvantages
16.4 An Example of a Delphi-Study on University Didactics Development in Germany
16.4.1 Preparation
16.4.1.1 Formation of Research Group and Development of Research Goals
16.4.1.2 Recruiting of the Sample
16.4.1.3 Designing the Study
16.4.2 The Four Delphi Rounds in More Detail
16.4.2.1 Round 1: Understanding Competence
16.4.2.2 Round 2: Selection
16.4.2.3 Round 3: Weighting
16.4.2.4 Round 4: Demand for Further Education
16.4.3 Reflection of Outcomes
16.5 Suggestions for Conducting Delphi-Studies
16.5.1 Research Group
16.5.2 Research Process
16.5.3 Composition of the Sample
16.5.4 Interaction with Participants
16.6 Conclusion
References
Chapter 17: The Narrative Approach to Research Professional Identity: Relational, Temporal, and Dialogical Perspectives
17.1 Introduction
17.2 Theoretical and Methodological Considerations in Narrative Identity Research
17.3 A Thematic Perspective on Professional Identity
17.3.1 Telling Professional Identity Narratives in Dialogical Interviews
17.3.2 Thematic Methods to Narrative Analysis
17.3.3 A Categorical-Thematic Analysis Aimed at Cultural Identity Narratives (Case 1)
17.3.4 A Holistic-Thematic Analysis: To Encompass Temporality in Identity (Case 2)
17.4 A Dialogical Approach to Narrative Self-Construction
17.4.1 Narrative Positioning (Case 3)
17.4.2 An Example of Narrative Positioning
17.5 Methodological Discussion on Narrative Professional Identity Research
17.6 Conclusions and Future Avenues: Narrative Identity and Learning Research
References
Chapter 18: Capturing Actions of Communities: Towards Virtual Ethnography and Digital Tools in Researching Organizations and W...
18.1 Introduction
18.2 Virtuality and Digital Tools in Ethnographic Research in Organizations and Communities
18.2.1 Background of Ethnographic Research on Organizations and Workplaces
18.2.2 Studying Workplace Learning in Virtual Environments
18.2.3 Subjective Evidence-Based Ethnography and the Study of Human Actions
18.3 Empirical Case Examples
18.3.1 Virtual Ethnography in Slack
18.3.1.1 Entering the Virtual Field
18.3.1.2 Glimpses of Learning Processes Through Events
18.3.2 A Subjective Evidence-Based Ethnography (SEBE) Study of Craft Practices
18.3.2.1 Setting Up SEBE
18.3.2.2 Analyzing SEBE Data
18.4 Concluding Thoughts
References
Chapter 19: Video-Based Interaction Analysis: A Research and Training Method to Understand Workplace Learning and Professional...
19.1 Introduction
19.2 Methodological Principles of Video-Based Interaction Analysis
19.2.1 Interaction Analysis as a Research Method
19.2.1.1 Origins and Definition
19.2.1.2 Theoretical Principles
19.2.1.3 Methodological Requirements
19.2.2 Collective Analysis of Interactional Data
19.2.2.1 Data Sessions as a Research Method
19.2.2.2 Objectives and Principles of Collective Data Sessions
19.2.3 Interaction Analysis as a Training Method
19.2.3.1 Empirical Exploration of Interaction Analysis in Training Settings
19.2.3.2 Methodological Principles of Interaction Analysis in Training
19.3 Implementing Video-Based Interaction Analysis in a Continuing Education Programme for Early Childhood Educators
19.3.1 Objectives and Empirical Research Design
19.3.2 Illustration and Case Study
19.4 Conclusion
Appendix
Transcription Conventions
References
Chapter 20: Q Method: Assessing Subjectivity Through Structured Ranking of Items
20.1 Introduction
20.2 A Primer on Q
20.2.1 A Brief Historical Note
20.2.2 The Why and How of Q Method
20.2.3 Four Steps of Using Q Method
20.2.4 Analysing and Interpreting Q-Sets
20.2.5 Assessing Advantages and Disadvantages of Q
20.3 Illustrating Q Method with an Empirical Study
20.3.1 Background and Purpose of the Study
20.3.2 Methods
20.3.2.1 Participants
20.3.2.2 Development of the Q-Set
20.3.2.3 Procedure
20.3.2.4 Data Analysis
20.3.3 Results
20.3.3.1 Perspective Pattern 1
20.3.3.2 Perspective Pattern 2
20.3.3.3 Perspective Pattern 3
20.3.3.4 Common Perspective Across All Participants
20.3.4 Discussion
20.4 Conclusion
References
Chapter 21: Eye Tracking in Professional Learning and Development: Uncovering Expertise Development Among Residents in Radiolo...
21.1 Introduction
21.2 Eye Tracking in Workplace Learning
21.3 Basic Understanding of Measuring Eye Movements
21.4 Empirical Illustration
21.5 Method
21.5.1 Design
21.5.2 Sample
21.5.3 Apparatus and Stimulus Material
21.5.4 Instrument
21.5.5 Procedure
21.5.6 Analysis
21.6 Results
21.6.1 Visual Information Processing During Clinical Reasoning: Residents´ Eye Movements
21.6.2 Residents´ Diagnostic Performance
21.7 Conclusion and Practical Implication
References
Chapter 22: Seeing Workplaces from a Social Network Analysis (SNA) Approach
22.1 Introduction
22.1.1 Workplaces and Communities-Whole Network Approach
22.1.2 Me and My Colleagues: Egocentric Approach to SNA
22.1.3 General Terms and Analyses in SNA
22.1.4 Data Gathering for SNA
22.1.5 Density and Centralisation-Most Typical Analyses in SNA
22.2 Case EcoTec Ltd.
22.2.1 Participants
22.2.2 Data Gathering
22.2.3 Density and Centralisation
22.2.4 Egocentric Approach and EcoTec Ltd.´s Key Workers
22.2.5 Visualisations
22.3 Discussion
22.3.1 Criticism of SNA and Some Limitations
22.3.2 Future Directions for SNA and Workplace Learning Studies
References
Chapter 23: Design-Based Research - Grounding, Understanding and Empirical Illustration in the Context of Vocational Education
23.1 Introduction or How Can Tablets Be Used in a Pedagogical Manner in Vocational Schools?
23.2 Characteristics of Design Research
23.3 Procedures of Design Research
23.4 Methods in Design-Based Research
23.5 Illustration of a Design-Based Research Project in Vocational Education
23.5.1 Context and Interest of the Study
23.5.2 Theoretical Framework of the tabletBS.dual Project
23.5.3 Realization of the Design-Based Research Approach
23.5.3.1 Design Phases in the Cycles
23.5.3.2 Evaluation Processes in the Design Cycles
23.6 Results of the Project
23.6.1 Results of the Digital Consultation Sessions
23.6.2 Results of the Webinars
23.6.3 Results of the Lesson Sequences Implemented with Tablets
23.7 Outlook
References
Chapter 24: Change Laboratory Method for Facilitating Transformative Agency and Collective Professional Learning - Case from a...
24.1 Introduction
24.2 The Change Laboratory Method
24.3 Conducting a Change Laboratory and Analysing CL Data
24.3.1 The Change Laboratory Meetings
24.3.2 Data Analysis
24.4 Findings: The Change Laboratory Facilitating Transformative Agency and Collective Professional Learning
24.4.1 Meeting 1 - Resisting and Criticising the Current Activity
24.4.2 Meeting 2 - Elaborating and Structuring the Challenges: Criticising and Explicating
24.4.3 Meeting 3 - Seeking a Shared Object Through Explicating and Envisioning
24.4.4 Meeting 4 - History Timeline Mediating the Development Efforts
24.4.5 Meeting 5 - Sharing Responsibility and Envisioning Through Practical Suggestions
24.4.6 Meeting 6 - Constructing a Model to Facilitate Educational Change
24.4.7 Follow-Up Meeting - Reflection of the Implementation
24.5 Discussion and Conclusion
References
Chapter 25: Professional Learning Analytics: Understanding Complex Learning Processes Through Measurement, Collection, Analysi...
25.1 What is Learning Analytics?
25.2 Learning Analytics as a Form of Automated Support for Learning
25.3 Learning Analytics Methods
25.4 Learning Analytics for Professional Learning
25.5 Case Studies from Professional Development MOOCs
25.5.1 What Data Is Available in MOOCs?
25.5.2 A Value Creation Framework for Multimodal Analytics
25.5.3 Tracing the Cycles of Value Creation with MMLA
25.5.4 Case Study 1: Analytics for Immediate Value: Designing Social Presence into a MOOC Platform
25.5.5 Case Study 2: Analytics for Applied Value: Promoting Peer Review
25.6 Conclusions
References
Chapter 26: Longitudinal Case Study Research to Study Self-Regulation of Professional Learning: Combining Observations and Sti...
26.1 Introduction
26.2 The Concept of Self-Regulation of Professional Learning
26.3 Measurement Challenges in Prevalent Methodological Paths
26.4 Proposing an Alternative Methodological Approach for SRpL Measurement
26.4.1 Why a Longitudinal Multiple Case Study Design Matters in Exploring Self-Regulation of Professional Learning
26.4.2 How Medical Specialists Self-Regulate Their Learning: Illustration of the Research Approach
26.4.2.1 Behaviours Under Observation
26.4.2.2 Time Aspects of the Observations
26.4.2.3 Performance of the Observations
26.4.2.4 Process and Structure of the Stimulated Recall Interviews
26.4.2.5 Data Preparation for Analysis
26.4.2.6 Analysis and Main Results
26.4.3 Opportunities and Challenges of the Proposed Methodology
26.4.3.1 Opportunities and Challenges in Data Collection
26.4.3.2 Opportunities and Challenges in Data Analysis
26.4.4 Suggestions for Future Research
26.5 Conclusion
References
Part IV: Discussion & Commentary
Chapter 27: Researching Professional Learning in Complex Environments: Opportunities and Challenges from a Qualitative Researc...
27.1 Introduction
27.2 The Contributions: Problems Addressed, Insights Generated
27.2.1 Studying PLD Through the Practices and Experiences of Individual Professionals
27.2.2 Accounting for Interactional Processes and Organisational Relations
27.2.3 Fostering and Responding to Change in Education and Work Environments
27.2.4 Summary
27.3 The Complexity of Work Environments: Emerging Challenges to Researching PLD
27.4 Conclusion
References
Chapter 28: How to Deal with the Complexity in Research on Workplace Learning
28.1 Introduction
28.2 How to Cope with the Complexity
28.3 Conclusion
References