This book highlights research on the behavioral biases affecting judgmental accuracy in judgmental forecasting and showcases the state-of-the-art in judgment-based predictive analytics. In recent years, technological advancements have made it possible to use predictive analytics to exploit highly complex (big) data resources. Consequently, modern forecasting methodologies are based on sophisticated algorithms from the domain of machine learning and deep learning. However, research shows that in the majority of industry contexts, human judgment remains an indispensable component of the managerial forecasting process. This book discusses ways in which decision-makers can address human behavioral issues in judgmental forecasting.
The book begins by introducing readers to the notion of human-machine interactions. This includes a look at the necessity of managerial judgment in situations where organizations commonly have algorithmic decision support models at their disposal. The remainder of the book is divided into three parts, with Part I focusing on the role of individual-level judgment in the design and utilization of algorithmic models. The respective chapters cover individual-level biases such as algorithm aversion, model selection criteria, model-judgment aggregation issues and implications for behavioral change. In turn, Part II addresses the role of collective judgments in predictive analytics. The chapters focus on issues related to talent spotting, performance-weighted aggregation, and the wisdom of timely crowds. Part III concludes the book by shedding light on the importance of contextual factors as critical determinants of forecasting performance. Its chapters discuss the usefulness of scenario analysis, the role of external factors in time series forecasting and introduce the idea of mindful organizing as an approach to creating more sustainable forecasting practices in organizations.
Author(s): Matthias Seifert
Series: International Series in Operations Research & Management Science, 343
Publisher: Springer
Year: 2023
Language: English
Pages: 319
City: Cham
Preface
Reference
Acknowledgments
Contents
Part I: Judgment in Human-Machine Interactions
Chapter 1: Beyond Algorithm Aversion in Human-Machine Decision-Making
1 Introduction
2 The Human vs. Machine Debate in Judgment and Decision-Making
3 Human-Machine Decision-Making
4 Beyond Algorithm Aversion: What Is Algorithm Misuse?
5 Causes of Algorithm Aversion and Algorithm Misuse
5.1 Prior Knowledge
5.2 Decision Control
5.3 Incentive Structures
5.4 Alignment of Decision-Making Processes
5.5 Alignment of Decision-Making Objectives
6 Towards Improved Methods and Metrics for Understanding and Resolving Algorithm Misuse
7 Conclusion
References
Chapter 2: Subjective Decisions in Developing Augmented Intelligence
1 Introduction
2 Theoretical Framework
2.1 Machine Learning-Based Augmented Reality
2.2 Design Science
2.3 Decision Making
3 Development Process
3.1 Use Case: Finding a Starting Point
3.2 MVP: Getting a First Version
3.2.1 Camera Feed
3.2.2 Execution Engine and Detection Model
3.2.3 Image Processing
3.2.4 Visualization
3.3 Summary of Steps 3-7: From a Proof of Concept to Future Use Cases
4 Decisions and Heuristics During the Development Process
4.1 Decision Types
4.1.1 Framework Decisions
4.1.2 Technological Decisions
4.1.3 Design Decisions
4.2 Decision Pyramids
4.2.1 Successive Decisions
4.2.2 Small and Large Worlds
4.3 Exemplary Development Decisions
4.3.1 General Environment
4.3.2 Framework Decisions
4.3.3 Technological Decisions
4.3.4 Design Decisions
5 Discussion
6 Limitations and Outlook
References
Chapter 3: Judgmental Selection of Forecasting Models (Reprint)
1 Introduction
2 Literature
2.1 Commonly Used Forecasting Models
2.2 Algorithmic Model Selection
2.3 Model Selection and Judgment
2.4 Combination and Aggregation
3 Design of the Behavioral Experiment
3.1 Selecting Models Judgmentally
3.2 Data
3.3 Participants
3.4 The Process of the Experiment
3.5 Measuring Forecasting Performance
4 Analysis
4.1 Individuals´ Performance
4.2 Effects of Individuals´ Skill and Time Series Properties
4.3 50% Statistics + 50% Judgment
4.4 Wisdom of Crowds
4.5 Evaluation Summary and Discussion
5 Implications for Theory, Practice, and Implementation
6 Conclusions
Appendix
Forecasting Models
Participants Details
References
Chapter 4: Effective Judgmental Forecasting in the Context of Fashion Products (Reprint)
1 Introduction
2 Theoretical Background
2.1 Judgment Analysis
2.2 Forecasting the Demand of Fashion Products
2.3 Hypotheses
3 Methods
4 Empirical Setting
5 Results
6 Discussion
References
Chapter 5: Judgmental Interventions and Behavioral Change
1 Background
2 The Design of a Behavioral Experiment
3 Results
4 Discussion
5 Conclusions
References
Part II: Judgment in Collective Forecasting
Chapter 6: Talent Spotting in Crowd Prediction
1 Introduction
1.1 Definition of Skill
1.2 Five Categories of Skill Correlates
2 Study 1
2.1 Study 1: Methods
2.1.1 Literature Search
2.1.2 Outcome Variables
2.1.3 Predictors of Skill
2.1.3.1 Accuracy-Related
2.1.3.2 Intersubjective
2.1.3.3 Behavioral
2.1.3.4 Dispositional
Fluid Intelligence and Related Measures
2.1.3.5 Expertise-Related
2.2 Study 1: Results
2.2.1 Accuracy-Related
2.2.2 Intersubjective
2.2.3 Behavioral
2.2.4 Dispositional
2.2.5 Expertise-Related
2.3 Study 1 Discussion
3 Study 2
3.1 Study 2: Methods
3.1.1 Good Judgment Project Data
3.1.2 Cross-Validation and Outcome Variable Definition
3.1.3 Predictor Selection
3.1.4 Statistical Tests
3.2 Study 2: Results
3.2.1 Correlational Analyses
3.2.1.1 Accuracy-Related Measures
3.2.1.2 Intersubjective Measures
3.2.1.3 Behavioral Measures
3.2.1.4 Dispositional Measures
3.2.1.5 Expertise Measures
3.2.2 Multivariate LASSO Models
3.3 Study 2: Discussion
4 General Discussion
4.1 Research Synthesis
4.2 Use Cases
4.3 Limitations and Future Directions
4.4 Conclusion
Appendix: Methodological Details of Selected Predictors
Item Response Theory Models
Contribution Scores
References
Chapter 7: Performance-Weighted Aggregation: Ferreting Out Wisdom Within the Crowd
1 Introduction
1.1 The Wisdom of Crowds
1.2 Judgment Quality: Defining and Identifying Expertise in the Crowd
2 Judgment Aggregation Strategies
2.1 Mean Strategies
2.2 Median Strategies
2.3 Weighting Functions
2.3.1 Weight All
2.3.2 Select Crowd
2.3.3 Hybrid Weighting Functions
2.4 Choosing a Weighting Function
3 Indicators of Expertise
3.1 History-Based Methods
3.1.1 Cooke´s Classical Method
3.1.2 Contribution Weighted Model
3.1.3 Discussion
3.2 Disposition-Based Methods
3.2.1 Domain Expertise
3.2.2 Psychometric Indicators of Individual Differences
3.2.3 Discussion
3.3 Coherence-Based Methods
3.3.1 Coherence Approximation Principle
3.3.2 Probabilistic Coherence Scale
3.3.3 Discussion
4 General Discussion
4.1 Ensemble Methods
4.2 Conclusion
References
Chapter 8: The Wisdom of Timely Crowds
1 Introduction
1.1 Forecaster Evaluation
1.2 Time Decay
1.3 Time and Crowd Size
2 Evaluating Forecasters Over Time
2.1 Forecast Timing
2.2 Information Accrual
2.3 Reliability of Forecaster Assessment
2.4 Recommendations
3 The Timeliness of Crowds
3.1 Selection Methods
3.2 Weighting Methods
3.3 Comparing Methods
3.4 A Probabilistic Hybrid Method
3.5 Martingale Violations
3.6 Recommendations
4 Crowd Size and Timing
4.1 Resampling the Crowd
5 General Discussion
5.1 Signal Sources
5.2 Bias
5.3 Beyond Judgmental Forecasting
5.4 Summary of Recommendations
5.4.1 Evaluating Forecasters
5.4.2 Information Accrual
5.4.3 Forecast Recency and Aggregation
5.4.4 Time and Crowd Size
References
Part III: Contextual Factors and Judgmental Performance
Chapter 9: Supporting Judgment in Predictive Analytics: Scenarios and Judgmental Forecasts
1 Introduction
2 Literature Review
3 Methodology
3.1 Experimental Design
3.1.1 Phase 1: Individual Forecasts
3.1.2 Phase 2: Team Forecasts with Scenario Discussions
3.1.3 Phase 3: Final/Preferred Individual Forecasts After Scenario Discussions
3.2 Results
3.2.1 Assessments of Scenario Tone
3.2.2 Individual Forecasts
3.2.3 Team Forecasts with Scenario Discussions
3.2.4 Final/Preferred Individual Forecasts After Scenario Discussions
4 Discussion
5 Conclusion
References
Chapter 10: Incorporating External Factors into Time Series Forecasts
1 Introduction
2 External Events
2.1 Event Characteristics
2.1.1 Magnitude and Duration
2.1.2 Regularity and Frequency
2.1.3 Predictability
2.2 Event Impact
2.2.1 Magnitude
2.2.2 Direction
2.2.3 Duration
2.2.4 Type
3 The Role of Judgment in Dealing with External Events
3.1 Judgmental Adjustment of Statistical Forecasts from Series Disrupted by External Events
3.2 Using Judgment to Select and Clean Data to Produce Baseline Forecasts
3.3 Judges´ Use of Analogical Strategies to Make Forecasts When Series Are Disrupted by External Events
4 Statistics to the Rescue?
4.1 Non-transparent Models
4.2 Transparent Models
5 Summary
References
Chapter 11: Forecasting in Organizations: Reinterpreting Collective Judgment Through Mindful Organizing
1 Introduction: Slow Progress Behind Paradigmatic Blinkers?
2 Showcasing the Effects of Functionalism in Forecasting Research
2.1 Extracting Forecasts from Groups
2.2 Learning from Feedback
3 Nuanced Organizational Aspects Towards a New Framework in Forecasting
3.1 Learning from Success Versus Failure
3.2 Group Deliberation About Performance
3.3 Team Leaders as Facilitators
4 Mindful Organizing: A Framework in the Interpretivist-Functionalist Transition Zone
5 Inducing Mindful Organizing to Debias Group Judgment
5.1 Focus on Episodic, Dramatic Error
5.2 Use of Analogical Reasoning and Reference Classes
6 Conclusion
References
Index
