Constructing Measures: An Item Response Modeling Approach

Cover
Endorsements
Half Title
Title
Dedication
Copyright
Contents
List of figures
List of tables
Preface
Aims of the Book
Audiences for the Book
Structure of the Book
Learning Tools
Using the Book to Teach a Course
Help for Old Friends
Acknowledgements
Part I A Constructive Approach to Measurement
1 The BEAR Assessment System: Overview of the “Four Building Blocks” Approach
1.1 What Is “Measurement”?
1.1.1 Construct Modeling
1.2 The BEAR Assessment System
1.3 The Construct Map
1.3.1 Example 1: The MoV Construct in the Data Modeling Assessments
1.4 The Items Design
1.4.1 Example 1: MoV Items
1.4.2 The Relationship between the Construct and the Responses
1.5 The Outcome Space
1.5.1 Example 1: The MoV Outcome Space
1.6 The Wright Map
1.6.1 Example 1: The MoV Wright Map
1.6.2 Return to the Discussion of Causation and Inference
1.7 Reporting the Results to the Measurer and Other Users
1.8 Using the Four Building Blocks to Develop an Instrument
1.9 Resources
1.10 Exercises and Activities
Part II The Four Building Blocks
2 Construct Maps
2.1 The Construct Map
2.2 Examples of Construct Maps
2.2.1 Example 1: The Models of Variability (MoV) Construct in the Data Modeling Curriculum
2.2.2 Example 2: A Social and Emotional Learning Example (RIS: The Researcher Identity Scale)
2.2.3 Example 3: An Attitude Example (GEB: General Ecological Behavior)
2.2.4 Example 4: A 21st Century Skills Example (LPS Argumentation)
2.2.5 Example 5: The Six Constructs in the Data Modeling Curriculum
2.2.6 Example 6: A Process Measurement Example—Collaborative Problem-Solving (CPS)
2.2.7 Example 7: A Health Assessment Example (PF-10: Physical Functioning 10)
2.2.8 Example 8: An Interview Example (CUE: Conceptual Underpinnings of Evolution)
2.2.9 Example 9: An Observational Instrument—Early Childhood (DRDP)
2.2.10 Example 10: The Issues Evidence and You (IEY) Science Assessment
2.3 Using Construct Mapping to Help Develop an Instrument
2.4 Examples of Other Construct Structures
2.5 Resources
2.6 Exercises and Activities
3 The Items Design
3.1 The Idea of an Item
3.2 The Facets of the Items Design
3.2.1 The Construct Facet
3.2.2 The Secondary Design Facets
3.3 Different Types of Item Responses
3.3.1 Participant Observation
3.3.2 Specifying (Just) the Topics
3.3.3 Constructed Response Items
3.3.4 Selected Response Items
3.3.5 Steps in Item Development
3.4 Building-in Fairness through Design
3.4.1 What Do We Mean by Fairness Here?
3.4.2 Universal Design
3.5 Resources
3.6 Exercises and Activities
4 The Outcome Space
4.1 The Qualities of an Outcome Space
4.1.1 Well-defined Categories
4.1.2 Research-based Categories
4.1.3 Context-specific Categories
4.1.4 Finite and Exhaustive Categories
4.1.5 Ordered Categories
4.2 Scoring the Outcome Space (the Scoring Guide)
4.3 General Approaches to Constructing an Outcome Space
4.3.1 Phenomenography
4.3.2 The SOLO Taxonomy
4.3.3 Guttman Items
4.4 A Unique Feature of Human Measurement: Listening to the Respondents
4.5 When Humans Become a Part of the Item: The Rater
4.6 Resources
4.7 Exercises and Activities
5 The Wright Map
5.1 Combining Two Approaches to Measurement
5.2 The Wright Map
5.2.1 The Rasch Model
5.2.2 Visualizing the Rasch Model Parameters: The Wright Map
5.2.3 Modeling the Response Vector
5.2.4 Linking the Construct Map and the Wright Map
5.3 The PF-10 Example (Example 7)
5.4 Reporting Measurements
5.4.1 Interpretation and Errors
5.4.2 The PF-10 Example (Example 7), Continued
5.5 Resources
5.6 Exercises and Activities
Textbox 5.1 Making sense of logits
Part III Quality Control Methods
6 Evaluating and Extending the Statistical Model
6.1 More Than Two Score Categories: Polytomous Data
6.1.1 The PF-10 Example (Example 7), Continued
6.2 Evaluating Fit
6.2.1 Item Fit
6.2.2 Respondent Fit
6.3 Resources
6.4 Exercises and Activities
Textbox 6.1 The Partial Credit Model
Textbox 6.2 Calculating the Thurstonian Thresholds
7 Trustworthiness, Precision, and Reliability
7.1 Trustworthiness in Measurement
7.2 Measurement Error: Precision
7.3 Summaries of Measurement Error
7.3.1 Internal Consistency Coefficients
7.3.2 Test–Retest Coefficients
7.3.3 Alternate Forms Coefficients
7.3.4 Other Reliability Coefficients and Indexes
7.4 Inter-rater Consistency
7.5 Resources
7.6 Exercises and Activities
8 Trustworthiness, Validity, and Fairness
8.1 Trustworthiness, Continued
8.1.1 Crafting a Full Validity Argument
8.2 Evidence Based on Instrument Content
8.2.1 Instrument Content Evidence for Example 2, the Researcher Identity Scale-G
8.3 Evidence Based on Response Processes
8.3.1 Response Process Evidence Related to Example 8—The DRDP
8.4 Evidence Based on Internal Structure
8.4.1 Evidence of Internal Structure at the Instrument Level: Dimensionality
8.4.2 Dimensionality Evidence for Example 2: The Researcher Identity Scale-G
8.4.3 Evidence of Internal Structure at the Instrument Level: The Wright Map
8.4.4 Wright Map Evidence from Example 2: The Researcher Identity Scale-G
8.4.5 Evidence of Internal Structure at the Item Level
8.4.6 Item-level Evidence of Internal Structure for the PF-10 Instrument
8.5 Evidence Regarding Relations to Other Variables
8.5.1 “Other Variables” Evidence from Two Examples
8.6 Evidence Based on the Consequences of Using the Instrument
8.7 Evidence Related to Fairness
8.7.1 Differential Item Functioning (DIF)
8.7.2 DIF Evidence for the RIS-G
8.8 Resources
8.9 Exercises and Activities
Part IV A Beginning Rather than a Conclusion
9 Building on the Building Blocks
9.1 Choosing the Statistical Model
9.1.1 Interpretation of Thurstone’s Requirement in Terms of the Construct Map
9.2 Comparing Overall Model Fit
9.3 Beyond the Lone Construct Map: Multidimensionality
9.4 Resources
9.5 Exercises and Activities
Textbox 9.1 Showing that Equation 9.5 Holds for the Rasch Model
Textbox 9.2 Statistical Formulation of the Multidimensional Partial Credit Model
10 Beyond the Building Blocks
10.1 Beyond the Construct Map: Learning Progressions
10.2 Beyond the Items Design and the Outcome Space: Process Measurement
10.3 Beyond the Statistical Model: Considering a More Complex Scientific Model
10.4 Other Measurement Frameworks: Principled Assessment Designs
10.4.1 Example: Evidence-centered Design
10.4.2 Going “Outside the Triangle”
10.5 A Beginning Rather Than a Conclusion
10.5.1 Further Reading about the History of Measurement in the Social Sciences
10.5.2 Further Reading about Alternative Approaches
10.5.3 Further Reading about the Philosophy of Measurement
10.6 Exercises and Activities
Appendix A The Examples Archive
Appendix B Computerized Design, Development, Delivery, Scoring and Reporting—BASS
Appendix C The BEAR Assessment System (BAS): Papers about its Uses and Applications
Appendix D Models of Variation Materials
Appendix E The General Ecological Behavior Items
Appendix F The Item Panel
Appendix G Matching Likert and Guttman Items in the RIS Example
Appendix H Sample Script for a Think-aloud Investigation
Appendix I The Item Pilot Investigation
Appendix J Results from the PF-10 Analyses
References
Index