Bayesian Networks in Educational Assessment

Cover
Statistics for Social and Behavioral Sciences
Bayesian Networks in Educational Assessment
Copyright
Springer Science+Business Media New York 2015
ISBN 978-1-4939-2124-9
ISBN 978-1-4939-2125-6 (eBook)
DOI 10.1007/978-1-4939-2125-6
Library of Congress Control Number: 2014958291
Dedication
Acknowledgements
Using This Book
Notation
Random Variables
Sets
Probability Distributions and Related Functions
Transcendental Functions
Usual Use of Letters for Indices
Contents
List of Figures
List of Tables


Part I Building Blocks for Bayesian Networks

1 Introduction
1.1 An Example Bayes Network
1.2 Cognitively Diagnostic Assessment
1.3 Cognitive and Psychometric Science
1.4 Ten Reasons for Considering Bayesian Networks
1.5 What Is in This Book

2 An Introduction to Evidence-Centered Design
2.1 Overview
2.2 Assessment as Evidentiary Argument
2.3 The Process of Design
2.4 Basic ECD Structures
2.4.1 The Conceptual Assessment Framework
2.4.2 Four-Process Architecture for Assessment Delivery
2.4.3 Pretesting and Calibration
2.5 Conclusion

3 Bayesian Probability and Statistics: a Review
3.1 Probability: Objective and Subjective
3.1.1 Objective Notions of Probability
3.1.2 Subjective Notions of Probability
3.1.3 Subjective–Objective Probability
3.2 Conditional Probability
3.3 Independence and Conditional Independence
3.3.1 Conditional Independence
3.3.2 Common Variable Dependence
3.3.3 Competing Explanations
3.4 Random Variables
3.4.1 The Probability Mass and Density Functions
3.4.2 Expectation and Variance
3.5 Bayesian Inference
3.5.1 Re-expressing Bayes Theorem
3.5.2 Bayesian Paradigm
3.5.3 Conjugacy
3.5.4 Sources for Priors
3.5.5 Noninformative Priors
3.5.6 Evidence-Centered Design and the Bayesian Paradigm

4 Basic Graph Theory and Graphical Models
4.1 Basic Graph Theory
4.1.1 Simple Undirected Graphs
4.1.2 Directed Graphs
4.1.3 Paths and Cycles
4.2 Factorization of the Joint Distribution
4.2.1 Directed Graph Representation
4.2.2 Factorization Hypergraphs
4.2.3 Undirected Graphical Representation
4.3 Separation and Conditional Independence
4.3.1 Separation and D-Separation
4.3.2 Reading Dependence and Independence from Graphs
4.3.3 Gibbs–Markov Equivalence Theorem
4.4 Edge Directions and Causality
4.5 Other Representations
4.5.1 Influence Diagrams
4.5.2 Structural Equation Models
4.5.3 Other Graphical Models

5 Efficient Calculations
5.1 Belief Updating with Two Variables
5.2 More Efficient Procedures for Chains and Trees
5.2.1 Propagation in Chains
5.2.2 Propagation in Trees
5.2.3 Virtual Evidence
5.3 Belief Updating in Multiply Connected Graphs
5.3.1 Updating in the Presence of Loops
5.3.2 Constructing a Junction Tree
5.3.3 Propagating Evidence Through a Junction Tree
5.4 Application to Assessment
5.4.1 Proficiency and Evidence Model Bayes Net Fragments
5.4.2 Junction Trees for Fragments
5.4.3 Calculation with Fragments
5.5 The Structure of a Test
5.5.1 The Q-Matrix for Assessments Using Only Discrete Items
5.5.2 The Q-Matrix for a Test Using Multi-observable Tasks
5.6 Alternative Computing Algorithms
5.6.1 Variants of the Propagation Algorithm
5.6.2 Dealing with Unfavorable Topologies

6 Some Example Networks
6.1 A Discrete IRT Model
6.1.1 General Features of the IRT Bayes Net
6.1.2 Inferences in the IRT Bayes Net
6.2 The ``Context'' Effect
6.3 Compensatory, Conjunctive, and Disjunctive Models
6.4 A Binary-Skills Measurement Model
6.4.1 The Domain of Mixed Number Subtraction
6.4.2 A Bayes Net Model for Mixed-Number Subtraction
6.4.3 Inferences from the Mixed-Number Subtraction Bayes Net
6.5 Discussion

7 Explanation and Test Construction
7.1 Simple Explanation Techniques
7.1.1 Node Coloring
7.1.2 Most Likely Scenario
7.2 Weight of Evidence
7.2.1 Evidence Balance Sheet
7.2.2 Evidence Flow Through the Graph
7.3 Activity Selection
7.3.1 Value of Information
7.3.2 Expected Weight of Evidence
7.3.3 Mutual Information
7.4 Test Construction
7.4.1 Computer Adaptive Testing
7.4.2 Critiquing
7.4.3 Fixed-Form Tests
7.5 Reliability and Assessment Information
7.5.1 Accuracy Matrix
7.5.2 Consistency Matrix
7.5.3 Expected Value Matrix
7.5.4 Weight of Evidence as Information


Part II Learning and Revising Models from Data

8 Parameters for Bayesian Network Models
8.1 Parameterizing a Graphical Model
8.2 Hyper-Markov Laws
8.3 The Conditional Multinomial—Hyper-Dirichlet Family
8.3.1 Beta-Binomial Family
8.3.2 Dirichlet-Multinomial Family
8.3.3 The Hyper-Dirichlet Law
8.4 Noisy-OR and Noisy-AND Models
8.4.1 Separable Influence
8.5 DiBello's Effective Theta Distributions
8.5.1 Mapping Parent Skills to Space
8.5.2 Combining Input Skills
8.5.3 Samejima's Graded Response Model
8.5.4 Normal Link Function
8.6 Eliciting Parameters and Laws
8.6.1 Eliciting Conditional Multinomial and Noisy-AND
8.6.2 Priors for DiBello's Effective Theta Distributions
8.6.3 Linguistic Priors

9 Learning in Models with Fixed Structure
9.1 Data, Models, and Plate Notation
9.1.1 Plate Notation
9.1.2 A Bayesian Framework for a Generic Measurement Model
9.1.3 Extension to Covariates
9.2 Techniques for Learning with Fixed Structure
9.2.1 Bayesian Inference for the General Measurement Model
9.2.2 Complete Data Tables
9.3 Latent Variables as Missing Data
9.4 The EM Algorithm
9.5 Markov Chain Monte Carlo Estimation
9.5.1 Gibbs Sampling
9.5.2 Properties of MCMC Estimation
9.5.3 The Metropolis–Hastings Algorithm
9.6 MCMC Estimation in Bayes Nets in Assessment
9.6.1 Initial Calibration
9.6.2 Online Calibration
9.7 Caution: MCMC and EM are Dangerous!

10 Critiquing and Learning Model Structure
10.1 Fit Indices Based on Prediction Accuracy
10.2 Posterior Predictive Checks
10.3 Graphical Methods
10.4 Differential Task Functioning
10.5 Model Comparison
10.5.1 The DIC Criterion
10.5.2 Prediction Criteria
10.6 Model Selection
10.6.1 Simple Search Strategies
10.6.2 Stochastic Search
10.6.3 Multiple Models
10.6.4 Priors Over Models
10.7 Equivalent Models and Causality
10.7.1 Edge Orientation
10.7.2 Unobserved Variables
10.7.3 Why Unsupervised Learning cannot Prove Causality
10.8 The ``True'' Model

11 An Illustrative Example
11.1 Representing the Cognitive Model
11.1.1 Representing the Cognitive Model as a Bayesian Network
11.1.2 Representing the Cognitive Model as a Bayesian Network
11.1.3 Higher-Level Structure of the Proficiency Model; i.e., p(bold0mu mumu [|bold0mu mumu [) and p(bold0mu mumu [)
11.1.4 High Level Structure of the Evidence Models; i.e., p()
11.1.5 Putting the Pieces Together
11.2 Calibrating the Model with Field Data
11.2.1 MCMC Estimation
11.2.2 Scoring
11.2.3 Online Calibration
11.3 Model Checking
11.3.1 Observable Characteristic Plots
11.3.2 Posterior Predictive Checks
11.4 Closing Comments


Part III Evidence-Centered Assessment Design

12 The Conceptual Assessment Framework
12.1 Phases of the Design Process and Evidentiary Arguments
12.1.1 Domain Analysis and Domain Modeling
12.1.2 Arguments and Claims
12.2 The Student Proficiency Model
12.2.1 Proficiency Variables
12.2.2 Relationships Among Proficiency Variables
12.2.3 Reporting Rules
12.3 Task Models
12.4 Evidence Models
12.4.1 Rules of Evidence (for Evidence Identification)
12.4.2 Statistical Models of Evidence (for Evidence Accumulation)
12.5 The Assembly Model
12.6 The Presentation Model
12.7 The Delivery Model
12.8 Putting It All Together

13 The Evidence Accumulation Process
13.1 The Four-Process Architecture
13.1.1 A Simple Example of the Four-Process Framework
13.2 Producing an Assessment
13.2.1 Tasks and Task Model Variables
13.2.2 Evidence Rules
13.2.3 Evidence Models, Links, and Calibration
13.3 Scoring
13.3.1 Basic Scoring Protocols
13.3.2 Adaptive Testing
13.3.3 Technical Considerations
13.3.4 Score Reports

14 Biomass: An Assessment of Science Standards
14.1 Design Goals
14.2 Designing Biomass
14.2.1 Reconceiving Standards
14.2.2 Defining Claims
14.2.3 Defining Evidence
14.3 The Biomass Conceptual Assessment Framework
14.3.1 The Proficiency Model
14.3.2 The Assembly Model
14.3.3 Task Models
14.3.4 Evidence Models
14.4 The Assessment Delivery Processes
14.4.1 Biomass Architecture
14.4.2 The Presentation Process
14.4.3 Evidence Identification
14.4.4 Evidence Accumulation
14.4.5 Activity Selection
14.4.6 The Task/Evidence Composite Library
14.4.7 Controlling the Flow of Information Among the Processes
14.5 Conclusion

15 The Biomass Measurement Model
15.1 Specifying Prior Distributions
15.1.1 Specification of Proficiency Variable Priors
15.1.2 Specification of Evidence Model Priors
15.1.3 Summary Statistics
15.2 Pilot Testing
15.2.1 A Convenience Sample
15.2.2 Item and other Exploratory Analyses
15.3 Updating Based on Pilot Test Data
15.3.1 Posterior Distributions
15.3.2 Some Observations on Model Fit
15.3.3 A Quick Validity Check
15.4 Conclusion

16 The Future of Bayesian Networks in Educational Assessment
16.1 Applications of Bayesian Networks
16.2 Extensions to the Basic Bayesian Network Model
16.2.1 Object-Oriented Bayes Nets
16.2.2 Dynamic Bayesian Networks
16.2.3 Assessment-Design Support
16.3 Connections with Instruction
16.3.1 Ubiquitous Assessment
16.4 Evidence-Centered Assessment Design and Validity
16.5 What We Still Do Not Know


A Bayesian Network Resources
A.1 Software
A.1.1 Bayesian Network Manipulation
A.1.2 Manual Construction of Bayesian Networks
A.1.3 Markov Chain Monte Carlo
A.2 Sample Bayesian Networks

References

Author Index

Subject Index