This volume provides a methodological toolbox for conducting policy research. Recognizing that policy research spans various academic disciplines, each of which takes a different view on causality, the volume introduces a methodologically pluralistic approach to policy studies. Each chapter clarifies the research question that each technique can answer, the research design and data treatment that each technique requires for its results to be sound, the validity domain of its results, and the actual deployment of the technique through a replicable example. Techniques covered include quasi-experimental designs, approaches to account for selection bias and observed imbalances, directed acyclic graphs and structural equation models, Qualitative Comparative Analysis, Bayesian case study and process tracing, and Agent-Based Modelling. By working through the volume, readers will understand how to learn from different techniques, apply them consciously, and triangulate them to make better sense of findings. This volume is intended for advanced academic courses, as well as scholars and practitioners in policy-related fields, such as political science, economics, sociology, and public administration.
This is an open access book.
Author(s): Alessia Damonte, Fedra Negri
Series: Texts in Quantitative Political Analysis
Publisher: Springer
Year: 2023
Language: English
Pages: 280
City: Cham
Preface
Acknowledgments
Contents
Chapter 1: Introduction: The Elephant of Causation and the Blind Sages
1.1 Policy Decisions and Causal Theories
1.2 The Elephant of Causation
1.2.1 Elephants by the Principle
1.2.2 Elephants by the Rules
1.2.2.1 Regularity
1.2.2.2 Counterfactual
1.3 The Blind Sages’ Portrayals as the Book’s Blueprint
1.3.1 Can this Single Factor Make Any Difference?
1.3.2 Through Which Structures?
1.3.3 Through Which Process?
1.3.4 Considerations and Extensions
References
Untitled
Chapter 2: Causation in the Social Realm
2.1 Why Discuss the Ontology of Causation?
2.2 Scientific Realism About the Social World and Social Causation
2.2.1 Critical Realism
2.3 What Is Causation?
2.3.1 Causal Mechanisms
2.3.2 Causal Powers
2.3.3 Manipulability and Invariance
2.4 Pluralism About Causal Inquiry
2.4.1 Case Studies and Process Tracing
2.4.2 Quantitative Research Based on Observational Data
2.4.3 Randomized Controlled Trials and Quasi-experimental Research
2.4.4 Generative Models and Simulation Methods
2.5 Realism and Methodological Pluralism
References
Suggested Readings
Chapter 3: Counterfactuals with Experimental and Quasi-Experimental Variation
3.1 Introduction
3.2 Causation and Counterfactual Impact Evaluation: The Jargon
3.2.1 Causes as Manipulable Treatments
3.2.2 Effects as Differences Between Factual and Counterfactual Outcomes
3.2.3 What the Data Tell (And When)
3.3 Shades of Validity
3.3.1 Internal Validity: The Ability to Make a Causal Claim from a Pattern Documented in the Data
3.3.2 Statistical Validity: Measuring Precisely the Relationship Between Causes and Outcomes in the Data
3.3.3 External Validity: The Ability to Extend Conclusions to a Larger Population, over Time and Across Contexts
3.4 Random Assignment Strengthens Internal Validity
3.5 Internally Valid Reasoning Without RCTs: Instrumental Variation
3.5.1 A Tale of Pervasive Manipulation
3.5.2 General Formulation of the Problem
3.5.3 Assumptions
3.5.3.1 The “Monotonicity” Assumption
3.5.3.2 The “As Good as Random” Assumption
3.5.3.3 The “Exclusion Restriction”
3.5.3.4 The “First-Stage” Requirement
3.5.4 Better LATE than Never
3.5.5 External Validity of Causal Conclusions
3.6 Causal Reasoning with Administrative Rules: The Case of Regression Discontinuity Designs
3.6.1 Larger Classes, Worse Outcomes?
3.6.2 Visual Interpretation
3.6.3 General Formulation of the Problem
3.6.3.1 The Sharp RD Design
3.6.3.2 The Fuzzy RD Design
3.6.4 Validating the Internal Validity of the Design
3.7 Conclusion
References
Suggested Readings
Chapter 4: Correlation Is Not Causation, Yet… Matching and Weighting for Better Counterfactuals
4.1 Introduction
4.2 Not Just a Mantra: Correlation Is Not Causation Because…
4.2.1 Causal Inference Entails an Identification Problem
4.2.2 Each Identification Strategy Entails a Set of Assumptions
4.2.3 Last but not Least: Model Dependence
4.3 Preprocessing Data with Matching to Improve the Credibility of the Estimates
4.3.1 No Magic: What Matching Can and Cannot Do
4.3.2 Useful Starting Point: Exact Matching
4.3.3 Propensity Score Tautology
4.3.4 How to Choose Among Matching Procedures?
4.3.5 The End: The Parametric Outcome Analysis
4.4 Empirical Illustration
4.4.1 Entropy Balancing
4.4.2 Coarsened Exact Matching
4.5 Conclusion
References
Suggested Readings
Chapter 5: Getting the Most Out of Surveys: Multilevel Regression and Poststratification
5.1 Introduction
5.2 How It Works
5.3 Running Example
5.3.1 Draw a Sample
5.3.1.1 Step 1: Fit a Model
5.3.1.2 Step 2: Construct the Poststratification Frame
5.3.1.3 Step 3: Predict and Poststratify
5.3.2 Beware Overfitting
5.3.3 Partial Pooling
5.3.4 Sample Size Is Critical
5.3.5 Stacked Regression and Poststratification (SRP)
5.3.6 Synthetic Poststratification
5.3.7 Best Performing
5.4 Conclusion
References
Chapter 6: Pathway Analysis, Causal Mediation, and the Identification of Causal Mechanisms
6.1 Introduction
6.2 Can Pathways Be Mechanisms?
6.3 Identifying Causal Mechanisms with Graphs
6.3.1 Closing the Backdoor
6.3.2 Closing the Front Door
6.4 Identifying Indirect Effects
6.4.1 Indirect Effect in Non-linear Systems
6.4.2 Indirect Effect When the Cause and the Mediator Interact
6.4.3 Wrapping Up
6.5 Applications
6.5.1 A Mechanistic View on the Worm Wars
6.5.2 A Mechanistic View on a Chicago School Reform
6.6 Thou Shall Not Raise Causal Illusions
References
Chapter 7: Testing Joint Sufficiency Twice: Explanatory Qualitative Comparative Analysis
7.1 Introduction
7.2 Interpretability
7.2.1 Mechanisms and Machines
7.2.2 Operationalizing Typological Theories
7.2.3 Assembling Configurational Hypotheses
7.3 Validity
7.3.1 QCA’s Algebra
7.3.1.1 Literals
7.3.1.2 Operators
Negation
Joint Occurrence
Alternatives
Necessity and Sufficiency
7.3.1.3 Truth Tables
7.3.2 Identifying Valid Inus Hypotheses
7.3.2.1 Rendering Hypotheses
7.3.2.2 Tackling Underspecification
Decision 1: Frequency Cut-Off
Decision 2: The Consistency Threshold
Decision 3: The Coverage Cut-Off
7.3.2.3 Tackling Overspecification
Irrelevant Components
A Note on Ambiguity in Solutions
Dealing with Trivial Factors
7.4 Soundness
7.4.1 Gauging for QCA: The Theoretical Side
7.4.1.1 The Starting Point
7.4.1.2 Ragin’s Reinvention
7.4.1.3 Fuzzy Sufficiency and Necessity
7.4.2 Gauging for QCA: The Empirical Side
7.4.2.1 Establishing the Universe of Reference
7.4.2.2 Operationalizing Intension
Hyper-Specificity
Hyper-Generality
The Problem of Missing Values
7.4.2.3 Identifying Membership Thresholds
7.5 Summing Up
References
Suggested Readings
Chapter 8: Causal Inference and Policy Evaluation from Case Studies Using Bayesian Process Tracing
8.1 Introduction
8.2 The Epistemic Foundations of Process Tracing
8.3 Process Tracing Best Practices and Examples from COVID Research
8.3.1 Definition of Process Tracing
8.3.2 How to Do Process Tracing
8.3.3 Best Practices in Process Tracing
8.3.3.1 Cast the Net Widely for Alternative Explanations
8.3.3.2 Be Equally Tough on the Alternative Explanations
8.3.3.3 Consider the Potential Biases of Evidentiary Sources
8.3.3.4 Consider Whether the Case Is Most or Least Likely for Alternative Explanations
8.3.3.5 Make a Justifiable Decision on When to Start
8.3.3.6 Be Relentless in Getting Diverse Evidence, but Make a Justifiable Decision on When to Stop
8.3.3.7 Combine PT with Case Comparisons if Relevant
8.3.3.8 Be Open to Inductive Insights
8.3.3.9 Use Deduction to Infer What Must Be True if a Hypothesis Is True
8.3.3.10 Remember Not All PT Is Conclusive
8.3.4 Examples from COVID Case Studies
8.4 The “Replication Crisis” and the Comparative Advantages of Process Tracing Case Studies
8.4.1 The Replication Crisis
8.4.2 Process Tracing on Complex Phenomena
8.4.3 Process Tracing in Multimethod Research
8.4.4 Process Tracing and Generalizing from Case Studies
8.4.5 Limitations of Process Tracing
8.5 New Developments in Process Tracing
8.5.1 Formal Bayesian Process Tracing
8.5.2 New Modes of Multimethod Research
8.6 Conclusions
References
Suggested Reading
Chapter 9: Exploring Interventions on Social Outcomes with In Silico, Agent-Based Experiments
9.1 Introduction
9.2 Agent-Based Modeling
9.3 Exploring Artificial Policy Scenarios
9.3.1 Interventions to Increase Competition or Collaboration in Science
9.3.1.1 Example 1
9.3.1.2 Example 2
9.4 Conclusions
References
Chapter 10: The Many Threats from Mechanistic Heterogeneity That Can Spoil Multimethod Research
10.1 Introduction
10.2 Basic Ideas Behind MMR
10.3 The Problem of Mechanistic Heterogeneity for External Validity in MMR
10.4 Sources of Mechanistic Heterogeneity in MMR
10.4.1 Complex Concepts or Measures
10.4.2 Known and Unknown Omitted Conditions
10.4.3 Causal and Temporal Dynamics
10.5 Taking Mechanistic Heterogeneity in MMR More Seriously
10.6 Concluding Remarks
References
Chapter 11: Conclusions. Causality Between Plurality and Unity
11.1 Introduction
11.2 Two Tales About the Making of Science
11.2.1 The Viewpoint of the History of Science
11.2.2 The Perspective of the Philosophy of Science
11.3 Can We Learn from One Another?
11.3.1 Ontological Incommensurability?
11.3.2 Epistemic Incommensurability?
11.3.3 Methodological Incommensurability?
11.3.3.1 Design-Based Solutions
11.3.3.2 Model-Based Solutions
11.4 Wrapping Up and Looking Ahead
References
