This book illustrates the latest developments in literacy and language assessment in the digital context, and subsequently presents a rigorous validation study on a newly proposed form of assessment (scenario-based assessment, SBA) that seeks to respond to the contextual change of literacy activities. It combines theories and innovative practices in both the literacy and language assessment sectors.
The empirical validation study on SBA, presented here, can help readers understand how digital scenarios can be realized in assessment practices with the aid of computer technology, and how the scenario settings in the digital context can affect EFL learners’ reading-to-writing performance. In this way, it can facilitate the reconceptualization of L2 literacy in the digital context. Moreover, the evidence and critical examination presented here can offer readers more comprehensive insights into the value or validity of a given innovative approach before it is adopted in their contexts.
Author(s): Yumei Zhang
Publisher: Springer
Year: 2022
Language: English
Pages: 194
City: Singapore
Acknowledgements
Contents
About the Author
Abbreviations
List of Figures
List of Tables
1 Introduction
1.1 Background
1.2 Purpose
1.3 Significance
1.4 Layout of the Book
References
2 Literature Review
2.1 Changes of Literacy Contexts Wrought by Information Technology
2.2 The Use of Computer Technology in Language Assessment
2.3 Scenario-Based Assessment
2.3.1 Definition and Key Features
2.3.2 Main Concerns Toward SBA
2.3.3 Previous Research on SBA
2.4 Validity Theories and Validation Methods
2.4.1 The Evolution of Validity Theories and Validation Frameworks
2.4.2 Argument-Based Approach to Validation
2.5 Reading-to-Write Literate Practice
2.5.1 Theoretical Frameworks
2.5.2 Previous Studies on the Reading-to-Write Ability
2.5.3 Argumentation as Reading-to-Write Practice
2.6 Summary of the Literature Review
References
3 Research Methodology
3.1 Research Questions
3.2 Theoretical Framework
3.3 Participants
3.3.1 Experts
3.3.2 Student Participants
3.3.3 Raters
3.4 Instruments
3.4.1 The SBRtW and RtW Tasks
3.4.2 Expert Judgment Guideline
3.4.3 Stimulated Recalls
3.4.4 Questionnaire
3.5 Procedures
3.5.1 Stage 1: Stimulated Recalls
3.5.2 Stage 2: A Quasi-Experiment
3.5.3 Stage 3: The Final Assessment
3.6 Data Analyses
3.6.1 Analyses of the Stimulated Recall Data
3.6.2 Preliminary Statistical Analyses
3.6.3 Repeated Measures ANOVA and Paired Sample t-Tests
3.6.4 Analyses Based on Generalizability Theory
3.6.5 Many-Facet Rasch Measurement
References
4 The Development of the SBRtW Assessment
4.1 Development Procedure and Guidelines
4.2 Purpose and Potential Use of the Test
4.3 The Scenarios and the Features
4.4 Topics
4.5 Reading Texts
4.6 Item Types and Formats
4.7 Scoring Rubrics for the Writing Tasks
4.7.1 Bottom-Up Extraction of the Holistic Rubric and Analytic Criteria
4.7.2 Rubric Refinement Informed by Theories and Previous Rubrics
4.7.3 Revision Based on Expert Judgment and Trial Scaling
References
5 Results
5.1 Appraisal of the Scoring Inference
5.1.1 The Appropriateness of the Scoring Rule
5.1.2 Rating Accuracy and Rater Agreement
5.2 Appraisal of the Generalization Inference
5.2.1 The Representativeness of the Observation Sample
5.2.2 Generalizability and Bias Analyses
5.3 Appraisal of Extrapolation Inference and Theory-Based Interpretations
5.3.1 The Authenticity of the SBRtW Assessment Tasks
5.3.2 The Underlying Processes in the SBRtW Assessment
5.3.3 The Usefulness of the SBA Features in Facilitating Performance
5.3.4 The Relationship Between Reading and Writing
5.3.5 The Progressiveness Across Grade Levels
5.4 Summary of the results
References
6 Discussion
6.1 Validity Argument for the Soring Inference
6.1.1 The Importance of Scoring Rubric to Construct Validity
6.1.2 The Role of Scoring Reliability in Validation
6.2 Validity Argument for the Generalization Inference
6.2.1 Task Sampling for Generalization
6.2.2 The Effect of Topical Issues
6.3 Validity Argument for Extrapolation and Theory-Based Interpretations
6.3.1 Task Authenticity for Validity
6.3.2 Underlying Processes as an Indication of Attributes at Play
6.3.3 The Effect of Scenario-Based Features on Writing Performance
6.3.4 The Role of Reading Comprehension in Writing Performance
6.3.5 Variance of Writing Performance Across Grade Levels
References
7 Conclusions
7.1 Summary of the Main Findings
7.2 Implications
7.2.1 Implications for Future Application of SBA
7.2.2 Implications for Digital Literacy Assessment
7.2.3 Implications for Technology Use in Language Assessment
7.3 Limitations and Suggestions for Future Research
References
Appendix A The Guidelines for Expert Judgments (Originally in Chinese)
Appendix B The Protocol for Stimulated Recall Procedure (English Version)
Appendix C The Questionnaire (Form 1 as an Example, Originally in Chinese)
Appendix D Informed Consent
知情同意书
Informed Consent (Translation)
Appendix E An Example of the SBRtW Assessment
Appendix F The Holistic Scale for the Writing
Appendix G The Analytic Rating Scale For Writing
Uncited References
