Virtual Standard Setting: Setting Cut Scores

Cover
Table of contents
List of figures
List of tables
List of acronyms
Chapter 1: Introduction
1.1 Overview of the study
1.2 Scope of the study
1.3 Outline of the chapters
Chapter 2: Literature review
2.1 Background to standard setting
2.2 The importance of setting valid cut scores
2.2.1 Standard setting methods
2.2.1.1 Examples of test-centred methods
Variants of the Angoff method
The Bookmark method
The Objective Standard Setting (OSS) method
2.2.1.2 Examples of examinee-centred methods
The Borderline Group (BG) method and the Contrasting Group (CG) method
The Body of Work (BoW) method
2.2.2 Evaluating and validating standard setting methods
2.3 Standard setting in language assessment
2.3.1 Current LTA standard setting research
2.3.1.1 The first publicly available CEFR alignment studies
2.3.1.2 Studies investigating understanding of method or CEFR
2.3.1.3 Studies investigating external validity evidence
2.3.1.4 Studies proposing new methods/modifications
2.4 Challenges associated with standard setting
2.4.1 Theoretical and practical challenges
2.4.2 Logistics
2.5 Virtual standard setting
2.5.1 Virtual standard setting: Empirical studies
2.5.2 Challenges associated with virtual standard setting
2.6 Media naturalness theory
2.6.1 Re-evaluating virtual standard setting studies through MNT
2.7 Summary
Chapter 3: Methodology
3.1 Research aim and questions
3.2 Methods
3.2.1 Embedded MMR design
3.2.2 Counterbalanced workshop design
3.2.3 Instruments
3.2.3.1 Web-conferencing platform and data collection platform
3.2.3.2 Test instrument
3.2.3.3 CEFR familiarisation verification activities
3.2.3.4 Recruiting participants
3.2.3.5 Workshop surveys
3.2.3.6 Focus group interviews
3.2.3.7 Ethical considerations
3.3 Standard setting methodology
3.3.1 Rationale for the Yes/No Angoff method
3.3.2 Pre-workshop platform training
3.3.3 In preparation for the virtual workshop
3.3.4 Description of the workshop stages
3.3.4.1 Introduction stage
3.3.4.2 Orientation stage
3.3.4.2.1 CEFR familiarisation verification activity A
3.3.4.2.2 CEFR familiarisation verification activity B
3.3.4.2.3 Familiarisation with the test instrument
3.3.4.3 Method training stage
3.3.4.4 Judgement stage
Round 1 Stage
Round 2 Stage
Round 3 Stage
3.4 Data analysis methods and frameworks
3.4.1 CEFR verification activities analysis
3.4.2 Internal validity of cut scores
Classical test theory (CTT)
Rasch measurement theory (RMT)
The many-facet Rasch measurement (MFRM) model
3.4.3 Comparability of virtual cut score measures
3.4.4 Differential severity
3.4.5 Survey analysis
3.4.6 Focus group interview analysis
3.6 Summary
Chapter 4: Cut score data analysis
4.1 Cut score internal validation: MFRM analysis
4.1.1 Rasch group level indices
4.1.2 Judge level indices
4.2 Cut score internal validation: CTT analysis
4.2.1 Consistency within the method
4.2.2 Intraparticipant consistency
4.2.3 Interparticipant consistency
4.2.4 Decision consistency and accuracy
The Livingston and Lewis method
The Standard Error method
4.3 Comparability of cut scores between media and environments
4.3.1 Comparability of virtual cut score measures
4.3.2 Comparability of virtual and F2F cut score measures
4.4 Differential severity between medium, judges, and panels
4.4.1 Differential judge functioning (DJF)
4.4.2 Differential medium functioning (DMF)
4.4.3 Differential group functioning (DGF)
4.5 Summary
Chapter 5: Survey data analysis
5.1 Survey instruments
5.2 Perception survey instrument
5.2.1 Evaluating the perception survey instruments
5.2.2 Analysis of perception survey items
Qualitative comments for communication item 1
Audio medium
Video medium
Qualitative comments for communication item 2
Audio medium
Video medium
Qualitative comments for communication item 3
Audio medium
Video medium
Qualitative comments for communication item 4
Qualitative comments for communication item 5
Audio
Video medium
Qualitative comments for communication item 6
Audio medium
Video medium
Qualitative comments for communication item 7
Audio medium
Video medium
Qualitative comments for communication item 8
Audio medium
Video medium
Qualitative comments for communication item 9
Audio medium
The video medium
5.3 Procedural survey items
5.3.1 Evaluating the procedural survey instruments
5.4 Summary
Chapter 6: Focus group interview data analysis
6.1 Analysis of transcripts
6.2 Findings
6.2.1 Psychological aspects
Distraction in the video medium
Self-consciousness in the video medium
Lack of non-verbal feedback in the audio medium
Inability to distinguish speaker in the audio medium
Inability to discern who was paying attention in audio medium
Cognitive strain in the audio medium
6.2.2 Interaction
Lack of small talk in virtual environments
No digression from the topic in virtual environments
Differences in amounts of discussion between virtual and F2F settings
6.2.3 Technical aspects
Technical problems in virtual environments
Turn-taking system
6.2.4 Convenience
Time saved in virtual environments
Freedom to multi-task in virtual environments
Less fatigue in virtual environments
6.2.5 Decision-making in virtual environments
6.3 Summary
Chapter 7: Integration and discussion of findings
7.1 Research questions
7.1.1 Research questions 1, 2, and 3
7.1.2 Research question 4
7.1.3 Research question 5
7.2 Limitations
7.3 Summary
Chapter 8: Implications, future research, and conclusion
8.1 Significance and contribution to the field
8.2 Guidance for conducting synchronous virtual cut score studies
Demands for facilitators and/or co-facilitators
Establishing a virtual standard setting netiquette
Selecting a suitable virtual platform
Selecting an appropriate medium for the workshop
Recruiting online participants
Training in the virtual platform
Uploading materials
Monitoring progress and engaging judges
8.3 Recommendations for future research
8.4 Concluding remarks
Appendices
Appendix A CEFR verification activity A (Key)
Appendix B Electronic consent form
Appendix C Judge background questionnaire
Appendix D Focus group protocol
Introductory statement
Focus group interview questions
Appendix E Facilitator’s virtual standard setting protocol
Appendix F CEFR familiarisation verification activity results
Appendix G: Facets specification file
Appendix H: Intraparticipant consistency indices
Appendix I: Group 5 group level and individual level Rasch indices
Appendix J: Form A & Form B score tables
Appendix K: DJF pairwise interactions
Appendix L: DGF pairwise interactions
Appendix M: Wright maps
References
Author index
Subject index
Series Index