Preface
Contents
About the Editors
Section Editors
Contributors
Part I: About This Handbook
1 Introduction to the Comparative Large-Scale Studies in Education: Structure and Overview of the Handbook
Part 1: Meta-Perspectives on International Large-Scale Assessments in Education
The Societal Role of International Large-Scale Assessments in Education
Theoretical Foundations of International Large-Scale Assessments in Education
Functions and Characteristics of International Large-Scale Assessments in Education
Accomplishments, Limitations, and Recommendations
Part 2: Methodology
Designing and Implementing ILSAs
Methods of Analysis
Potential and Methods of Linking ILSA to National Education Policy and Research
Part 3: Findings
Schools, Principals, and Institutions
Classrooms, Teachers, and Curricula
Students, Competences, and Dispositions
Equity and Diversity
2 Background, Aims, and Theories of the Comparative Large-Scale Studies in Education
Introduction
Aim
Themes and Content
The Theoretical Foundation and Perspectives of the Handbook
Educational Effectiveness
The Context-Input-Process Output Model
The Hierarchical School System
Delineating the Scope
Perspectives and Principles Underlying the Handbook
Politically and Culturally Balanced
Methodological Perspectives
Extended Student Outcomes
Positioning the Handbook Within the Existing Literature
The Audience of This Handbook
Summary
References
Part II: Meta-perspectives on ILSAs: Theoretical Meta-perspectives on ILSAs
3 The Political Economy of ILSAs in Education: The Role of Knowledge Capital in Economic Growth
Overview
Understanding Growth
A Conceptual Framework for Knowledge and Growth
Growth Models with School Attainment
An Extended View of the Measurement of Human Capital
Knowledge Capital and Growth
Causality
The Gains from Universal Basic Skills
The Global Challenge
Economic Impacts of Universal Basic Skills
Conclusions
Cross-References
References
4 Reasons for Participation in International Large-Scale Assessments
Introduction
The Appeal of International Large-Scale Student Assessments
Creating a Demand for ILSAs
Conclusion
References
5 Educational Accountability and the Role of International Large-Scale Assessments
Introduction
Evidence-Based Educational Policymaking and Outcomes-Based Accountability
A Brief Overview of the History and Purpose of ILSAs
Administrative Accountability and the Role of ILSAs
A Framework for Considering the Potential for ILSAs in Accountability Systems
Research on Administrative Accountability and the Role of ILSAs
Conclusion
References
6 International Large-Scale Assessments and Education System Reform
Introduction
The Rise of International Large-Scale Assessments
The Power of Numbers
Changing Cognition and Behavior
Changing Behaviour
Penetrating Schools and Classrooms: PISA for Schools
An Alternative Model: The OECD´s SEG Framework
Case Studies
Learning Seminars
A Holistic Approach to International Comparisons?
Conclusions
References
7 The Role of International Large-Scale Assessments (ILSAs) in Economically Developing Countries
Chapter Summary
Background
Evidence-Based Policy-Making
Factors Influencing the Use of Evidence in Policy-Making
A Special Case of ILSAs: Two Regional Large-Scale Assessment Programs
Pacific Islands Literacy and Numeracy Assessment (PILNA)
A Brief Overview of PILNA
How Is PILNA Integrated into the Policy-Making Process of Participating Countries?
What Is the Role of Capacity Building Activities and Technical Quality in PILNA?
Research Capacity and the Use of Data
Quality of the Assessment Program
What Does PILNA´s Strategy Look like in Terms of Access and Results Dissemination to Stakeholders, Including the Media?
Southeast Asia Primary Learning Metrics (SEA-PLM)
A Brief Overview of SEA-PLM
What Is the Role of Capacity Building Activities and Technical Quality in SEA-PLM?
How Is SEA-PLM Integrated into the Policy-Making Process of Participating Countries?
What Does SEA-PLM´s Strategy Look like in Terms of Access and Results Dissemination to Stakeholders, Including the Media?
Conclusion
References
Part III: Meta-perspectives on ILSAs: The Role of Theory in ILSAs
8 Comprehensive Frameworks of School Learning in ILSAs
Introduction
Models of School Learning and their Developments
The Shift from Input-Output to Input-Process-Output Paradigm
Extensions from the Socio-Ecological Perspective
Extensions Reflecting the Dynamic Perspectives
Models of School Learning in ILSA Frameworks
The Beginning: Pilot Twelve-Country Study, Early ILSAs (FIMS, SIMS, IRLS), and IEA´s Foundational Curriculum Model
Pilot Twelve-Country Study (1960)
FIMS: First International Mathematics Study (1964)
Six-Subject Study and FISS (First International Science Study; 1970-71)
SIMS: Second International Mathematics Study (1980-82) and SISS: Second International Science Study (1983-84)
Subsequent ILSAs: IEA´s TIMSS and PIRLS and OECD´s PISA and TALIS
TIMSS: Third International Mathematics and Science Study (1995) and Trends in International Mathematics and Science Study (4-Y...
IRLS: International Reading Literacy Study (1990-91) and PIRLS: Progress in International Reading Literacy Study (5-Year Cycle...
PISA: Programme for International Student Assessment (3-Year Cycle from 2000 on)
TALIS: Teaching and Learning International Survey (5-Year Cycle from 2008 on)
Early Childhood Education and Care (ECEC)
Preprimary Project (1987-1989, 1992, 1995-97) and the IEA ECES: Early Childhood Education Study (2015)
OECD Starting Strong Survey (2018)
Conclusions and Recommendations
References
9 Assessing Cognitive Outcomes of Schooling
Introduction
Conceptualization of Cognitive Outcomes
Similarities and Differences of Theoretical Frameworks Across Different ILSA Studies
Organization of Theoretical Frameworks
Comparison of Content Dimensions
Comparison of Cognitive Dimensions
Development of Cognitive Dimension Across ILSAs Over Time
Mathematics Assessment Frameworks
IEA Studies
PISA
Science Assessment Frameworks
IEA Studies
PISA
Reading Assessment Frameworks
IEA PIRLS
PISA
Implementation of the Frameworks in Item Design
Format of Assessment
Comparison of TIMSS and PISA Items´ Features
New Forms of Assessment
Discussions and Conclusion
Summary of Findings
Implications
The Design of Future ILSAs
What Should a Country Be Achieving in Education?
Concluding Remarks
References
10 Socioeconomic Inequality in Achievement
Introduction
Distributive Rules
Different Goods, Different Rules
Socioeconomic Inequality: Implicit Assumptions
Measurement Issues
Indicators of Socioeconomic Status
Classification of Continuous and Categorical Measures
Standardization and Threshold Setting: International Comparability and National Specificity
Empirical Analyses
Data and Variables
Correlations Between the Different Measures of SES
Correlations Between the Different Measures of SES Inequality in Achievement
Standardization of Inequality Measures: Relative and Absolute Measures
Concluding Remarks
Cross-References
References
11 Measures of Opportunity to Learn Mathematics in PISA and TIMSS: Can We Be Sure that They Measure What They Are Supposed to ...
Research Problem
Previous Research on OTL Effects
Conceptual Framework
The Validity of OTL Measures
Research Approach and Research Questions
Method
Assessment of Content Validity
Assessment of Convergent Validity
Results
Content Validity: Comparison of the Content of the TIMSS and PISA OTL Measures
OTL Measures in TIMSS and PISA
Assessment/Analytical Frameworks
Similarities Between the Assessment/Analytical Frameworks and the OTL Measures
Key Findings: Interpretation of the Content Comparison with an Eye to Content Validity and Concurrent Validity
Interpretation of the Comparison in Terms of Face Validity
Convergent Validity: The Association of OTL with Student Performance in TIMSS and PISA
Data Analysis
Key Findings with Regard to Convergent Validity
Discussion
Conclusion
References
12 Using ILSAs to Promote Quality and Equity in Education: The Contribution of the Dynamic Model of Educational Effectiveness
Introduction
A Brief History of International Large-Scale Assessment Studies in Education
ILSAs Conducted by the IEA
ILSAs Carried Out by the OECD Measuring Student Learning Outcomes
A Brief Historical Overview of the Educational Effectiveness Research
Connections Between ILSA and EER
The Dynamic Model of Educational Effectiveness: An Overview
Advancements of ILSAs by Making Use of the Dynamic Model
References
Part IV: Meta-perspectives on ILSAs: Characteristics of ILSAs
13 Overview of ILSAs and Aspects of Data Reuse
Introduction
Agencies, Studies and Cycles, and Participating Entities
ILSA Objectives
Domains of Investigation
Target Populations and Samples
Data Collection
ILSA International Databases and Aspects Related to Data Analysis
Data Reuse (Secondary Analyses)
Conclusion
References
14 IEA´s TIMSS and PIRLS: Measuring Long-Term Trends in Student Achievement
IEA´s TIMSS and PIRLS: Measuring Long-Term Trends in Student Achievement
Gradually Updating the TIMSS and PIRLS Frameworks
Role of the TIMSS and PIRLS Encyclopedias
Roles of Committees of International Experts
Role of the National Research Coordinators
Large Numbers of Items Continually Refreshed
The PIRLS 2021 Rotated Design and Number of Items
The TIMSS 2023 Rotated Design and Number of Items
Introducing Ambitious Innovations on a Small Scale
Less Difficult TIMSS and PIRLS Assessments
Transitioning to e-Assessment
Methodology for Linking Successive Assessments to a Common Scale
Item Calibration, Conditioning, and Plausible Values
Concurrent Calibration
Bridges When There Are Major Changes
TIMSS 2007 Bridge
TIMSS 2019 Bridge
Phase 1: Item Calibration for paperTIMSS and Bridge Data
Phase 2: Linking eTIMSS Data to TIMSS Trend Scales
Conclusion/Summary
References
15 IEA´s Teacher Education and Development Study in Mathematics (TEDS-M)
Introduction
The Need for a Teacher ILSA
Structure of the Present Chapter
Conceptual Framework
Macro Level: Countries´ National Context
Meso Level: Institutional Context
Microlevel: Future Teachers´ Background
Teacher Education Outcomes: Teachers´ Professional Knowledge
Teacher Education Outcomes: Teachers´ Professional Beliefs
Research Questions
Study Design of TEDS-M
Sampling
Instruments: National (Macro Level) and Institutional Context (Meso Level)
Instruments: Future Teacher Surveys (Microlevel)
Data Analysis
Results
Macro Level: Structure and Quality Assurance of Teacher Education
Meso Level: Characteristics of Teacher Educators and Opportunities to Learn
Microlevel: Characteristics of Future Teachers (Based on Blömeke and Kaiser, 2014)
Outcomes of Teacher Education
Further Developments
Conclusions
References
Useful Resources
16 OECD Studies and the Case of PISA, PIAAC, and TALIS
Introduction
Programme for International Student Assessment (PISA)
Survey Objectives
Survey Periodicity and Country/Economy Participation
Survey Design, Methods, and Instruments
Assessment Framework
How PISA Differs from Other International Large-Scale Assessments
Key Insights to Date
Almost One in Four Students Did Not Reach a Minimum Proficiency Level in Reading on Average Across OECD Countries in 2018
Pulling Up Low Performers Without Affecting Overall Performance or Top Performers
PISA Results Help Identify Low Performers and Where They Are
Aiming for High Performance and Equity in Education Simultaneously
Allocation of Resources Matters More Than the Level of Spending After a Certain Investment Threshold
Identifying the Conditions That Support the Positive Relationship Between School Autonomy and Students´ Performance
High-Performing Education Systems Hold High Expectations for Their Students
Students in High-Performing Education Systems Tend to Believe Their Abilities Can Be Developed Over Time
Future Developments
The Survey of Adult Skills (PIAAC)
Survey Objectives
Survey Periodicity and Country Participation
Survey Design and Methods
Study Instruments
Background Questionnaire
Direct Assessment
Assessment Frameworks
Relationship of PIAAC to Previous International Adult Assessments
Key Insights
Developments in the Second PIAAC Assessment
The Teaching and Learning International Survey (TALIS)
Survey Objectives
Survey Periodicity and Country/Economy Participation
Survey Design and Methods
Study Instruments
Survey Conceptual Framework
Relationship of TALIS to Other International Surveys
Key Insights
A Key Asset for Many Countries Is to Benefit from a Highly Educated and Altruistic Teacher Workforce
Yet, the Delivery of Quality Instruction Is Often Hindered by Shortages of Qualified Teachers, and Novice Teachers Tend to Be ...
This Underlines the Importance of Induction for a Flying Start in the Profession
But Exchanges and Collaboration with Peers Matter Beyond the Early Years and Throughout Teachers´ Careers
The School and Classroom Climate and Culture also Matter
Teachers´ Perceptions About Their Jobs Are Related to a Range of Different Factors
Appraisal Systems, if Well-Designed, Hold Great Potential
And School Leaders Have a Role to Play to Foster a Greater Sense of Job Satisfaction Among Teachers
Future Development Perspectives
Conclusion
References
17 Regional Studies in Non-Western Countries, and the Case of SACMEQ
Introduction
Main Aims, Frameworks, and Designs of ILSA Studies
Main Aims
Frameworks of ILSAs
Organizing Framework Proposed by the OECD
Conceptual Frameworks of the IEA
Designs
Frequency and Scale of Studies
Population and Sampling
SACMEQ Studies
Introduction
Aims and Objectives of SACMEQ
Governance, Management, and Coordination of SACMEQ
Framework of SACMEQ
Design of SACMEQ Studies I-IV
Population and Sampling
Instruments
Tests
Questionnaires
Data Collection and Analysis
Data Collection
Analysis
Questionnaire Analysis
Reflections and Conclusion
Cross-References
References
18 The Use of Video Capturing in International Large-Scale Assessment Studies: Methodological and Theoretical Considerations
Introduction
Issues and Challenges to Consider
Purpose of the Study
Dimensions of Teaching Practices Captured
Dimensions of Teaching Practices Captured: Looking Across Frameworks
Theoretical Underpinning: Views of Teaching and Learning
Subject Specificity
Grain Size
Scoring Specifications
Focus on Students or Teachers (or Both)
Empirical Evidence: Connecting Teaching with Student Outcomes
Ethics
Technology: A New Generation of Video Studies
Moving Forward
Teacher Evaluation
Video Documentaries As Longitudinal Data
Videos As a Means of Improving Teachers´ Professional Learning
Concluding Remarks
References
19 Comparison of Studies: Comparing Design and Constructs, Aligning Measures, Integrating Data, Cross-validating Findings
Introduction
Prelude: Covering 48 years in Mathematics Education
Comparing and Finding Common Grounds in Design and Conceptualization
Country Coverage
Sampling
Domain of Assessment and Further Skills Covered
Context, Input, Process, and Output-Related Constructs
Comparing, Linking, and Matching Operational Measures
Differences in Instruments and Measures
Linking Measures from Different Studies
Searching for Universal Descriptors and Scales
Combining Data Across Assessments in Educational Research
Integrating Diverging and Converging Findings Across Studies
Matching Data
Longitudinal Designs
Analyzing ILSA Data on the Country Level
Comparing TIMSS and PISA Achievement Results on the Country Level
Comparing Country Level Trends Based on TIMSS and PISA
Conclusions and Implications for Further Research
References
Part V: Meta-perspectives on ILSAs: Accomplishments, Limitations, and Recommendations
20 ILSA in Arts Education: The Effect of Drama on Competences
Introduction
Historical, Pedagogical, and Epistemological Grounding
Concepts of Knowledge and Arts Education
The Field of Drama/Theatre Education
Drama as an Aesthetic Discipline
Drama as an Arts Education Subject
Learning About and In the Drama Subject: The Double Content
Learning With or Through Drama: Drama as a Learning Method in Other Subjects
Drama - and the Arts Hegemony in Schools
Applied Drama and Theatre
Competences Which Develop Through the Arts
An ILSA in Drama Education: The DICE Study
Motivation, Consortium, Objectives, and Hypothesis
An Introduction to the Methodology
The Sample
Data Collection
Eight Different Sources of Data
Overview of the Key Results
Discussion: How Arts Educators Assess Assessments
Lack of International Large-Scale Quantitative Assessments in the Field of Drama Education
Scepticism of Drama Researchers and Practitioners About Quantitative Measurements
Conclusion: Designing Large-Scale Assessment Studies in Drama Education
Appendix
References
21 Future Directions, Recommendations, and Potential Developments of ILSA
Preliminary Remarks
Introduction
History
Increased Participation
Future Directions
Recommendations
Increased Coverage
Study Frameworks
Changes in Curricula
Future Directions
Recommendations
Increased Depth: Added Value Through Computer-Based Assessment
Future Directions
Recommendations
Summary
References
22 Conceptual and Methodological Accomplishments of ILSAs, Remaining Criticism and Limitations
ILSAs: Products of Their Time
A Brief Summary of ILSA History
Conceptual Accomplishments and Limitations
Boosting Theory Development in Education
Comprehensive Conceptual Modelling of Educational Outcomes
Including Educational Outcomes Beyond Cognition
Including Cognitive Outcomes Beyond Mathematics, Science, and Reading
Including Educational Outcomes of Populations Beyond School Students
Including Educational Outcomes of Low- and Middle-Income Countries
Describing Trends in Educational Outcomes and Boosting Longitudinal Studies
Modelling Predictors of Educational Outcomes
Limited Theoretical Foundation Jeopardizing Validity
Definition and Operationalization of Constructs
What Do We Want and Need to Measure?
How Do We Want to Compare the Results?
Comparability of Items and Constructs Across Countries
Comparability of Target Populations Across Countries
What Is Driving the Implementation and Expansion of ILSAs?
Methodological Accomplishments and Limitations
Infrastructure and Researcher Training: Boosting Assessment Capacity
Strengthening Measurement Quality
Further Development of Technical Approaches
Implementing New Types of Assessments
Challenges to Ensure Measurement Quality in Other Respects
Limited Estimation Precision for Highest and Lowest Performing Countries
Limited Comparability Over Time Due to Assessment Changes
Limited Accounting for the Cluster Structure
Lack of Possibility to Draw Causal Inferences
Difficulties to Communicate Methodology and Study Limitations
Empirical Accomplishments and Limitations
Setting Education and Schooling Back on the Agenda
Making Sure That Public Education Has the Quality Needed
Policy, Media, and ILSAs: Sometimes an Unfortunate Combination
Summary and Conclusions
Summary
Conclusions
References
Part VI: Methodology: Designing and Implementing ILSAs
23 Sampling Design in ILSA
Introduction
Ensuring Cross-National Comparability
Inferring Population Parameters from Complex Samples
Sampling Variance in Complex Samples
Summary and Conclusions
Cross-References
References
24 Designing Measurement for All Students in ILSAs
Introduction
Expanding the Scope in ILSA Background Questionnaires
Expanding Achievement Estimation in ILSAs
Discussion
Conclusion
References
25 Implementing ILSAs
Introduction
Comparability, Validity, and Reliability of International Assessments
International Large-Scale Assessments Discussed in This Chapter
PISA
PIRLS
TIMSS
ICILS
ICCS
Identifying a Common Construct Domain
Process and Procedures of Defining and Operationalizing a Domain
Comparability and Cross-Cultural Validity
Other Sources of Incomparability in International Assessments
Supervision and Quality Assurance
The Role of Technology and Complexity
The Widening Participation of Middle-Income Countries
Validation of Innovative Constructs in International Assessment
Financial Literacy (PISA)
Problem Solving (PISA)
Problem Solving and Inquiry (TIMSS)
Civic Knowledge (ICCS)
Conclusion
References
26 Dilemmas in Developing Context Questionnaires for International Large-Scale Assessments
Introduction
Overview of the Questionnaire Development Process
Updating the TIMSS, PIRLS, and PISA questionnaires
Overarching Principles for Questionnaire Development
Questionnaire Development Dilemmas
Dilemma 1: Use of scales and indices versus stand-alone items to measure educational constructs
Dilemma 2: Updating the Surveys Versus Measuring Trends
Dilemma 3: Prioritizing predictors of achievement scores versus conceptualizing questionnaire items as stand-alone outcomes
Dilemma 4: Covering versus not covering topical policy topics
Dilemma 5: Including innovative item types versus all traditional items
Conclusions
References
27 Multistage Test Design Considerations in International Large-Scale Assessments of Educational Achievement
Introduction
Some Design Choices and a Review of Related Studies
Realizing the Potential of MST
Conclusion
Appendix A: Some Technical Details for SLRR19 and RLSR20
RLSR20
SLRR19
References
Part VII: Methodology: Methods of Analysis
28 Secondary Analysis of Large-scale Assessment Databases
Introduction
About Large-scale Assessments
Sample Size and Reporting Requirements in Large-scale Assessments
Uncertainty in Large-scale Assessments
What Are Plausible Values?
How to Work with Plausible Values?
How to Compute Measurement Variance?
What Are Sampling Weights?
How to Work with Sampling Weights?
How to Compute Sampling Variance?
Using Replicate Weights
Taylor Series Approximation
Why Do These Things Matter?
References
29 Methods of Causal Analysis with ILSA Data
Introduction
Causality
Correlation and Causation
The Randomized Experiment
An Example: The STAR Experiment
The Endogeneity Problem
Models of Causality
Endogeneity Bias and Other Threats to Valid Causal Inference
Analytical Techniques to Support Causal Inference from Observational Data
Fixed Effects Models
Fixed Effects Models: An Illustration of the Age Effect on Mathematics Achievement
Fixed Effects Models: Alternative Techniques for Conducting Analyses with Fixed Country Effects
Regression Discontinuity Designs
Regression Discontinuity Designs: An Empirical Example of the Schooling Effect
Instrumental Variable Techniques
IV Techniques: Correcting for Errors of Measurement in Independent Variables
IV Techniques: Estimating IV Models with SEM
IV Techniques: Estimating Fuzzy Regression
Conditioning on Observed and Latent Variables
Discussion and Conclusions
Criticisms of Causal Approaches in ILSA Research
How to Deal with the Endogeneity Problem?
References
30 Trend Analysis with International Large-Scale Assessments
Introduction
Brief Background and Design of TIMSS, PIRLS, and PISA
How Are Trends Reported for TIMSS and PIRLS?
How Are Trends Reported for PISA?
Reporting of Trend Results and Policy Reactions
Current Issues in Trend Reporting
Future Directions and Opportunities
References
31 Cross-Cultural Comparability of Latent Constructs in ILSAs
The Framework of Bias and Equivalence
Taxonomy of Bias
Taxonomy of Equivalence
Review of the Practice of Equivalence Testing in ILSAs
General Remarks
Scale Reliability by Country
PCA by Country
Comparison of Scales´ Correlations by Country
CFA Model Fit for Each Country
MGCFA
IRT Item Fit
Model-Based Approaches for Partial Measurement Invariance
The Expanding Toolbox of Equivalence Testing
Conclusions and Recommendations
References
32 Analyzing International Large-Scale Assessment Data with a Hierarchical Approach
Introduction
Aims of This Chapter
Systematic Review of Studies Taking a Hierarchical Approach to Analyzing ILSA Data
Context and Aims of the Systematic Review
Literature Search
Screening and Coding of Studies
Description of the Selected Studies
Publication Features
Data Structures
Aggregation of Variables and Construct Representation
Centering of Predictor Variables
Model Estimation and Evaluation
Handling Missing Data
Review of Selected Multilevel Modeling Approaches
Brief Overview of Existing Frameworks
Multilevel Regression Models
The Null Model
The Hierarchical Linear Model
Contextual Effects Models
Cross-Level Interaction Models
Limitations of Multilevel Regression Analysis
Multilevel Structural Equation Models
Latent Decomposition of Observed Variables
Latent Covariate Contextual Models
Multilevel Mediation Models
Multilevel Confirmatory Factor Analysis
Structural Models in MSEM
Multilevel Mixture Models
Issues Specific to the Multilevel Modeling of ILSA Data
Plausible Values
Multigroup and Incidental Multilevel Data Structures
Survey Weights
Illustrative Examples
Data Sources
Results
Multilevel Regression Models
Multilevel Confirmatory Factor Analysis
Multilevel Structural Equation Models
Multilevel Latent Profile Analysis
Conclusion
Appendices
Appendix A: Database Search Terms
Appendix B: PRISMA Statement
Appendix C: Description of the Variables Used in the Illustrative Examples
References
33 Process Data Analysis in ILSAs
Introduction
What Are Process Data?
What Kind of Information Can We Get from Process Data?
How Are Process Data Related to Response Process?
An Ecological Framework for the Analysis of Process Data
How Could the Proposed Ecological Framework Help the Analysis of Process Data from ILSAs?
Method
What Do We Know About the Analysis of Process Data from ILSAs?
Which Studies with Process Data from ILSAs Are Analyzed Here?
Results
When Did the First Studies Start to Be Published?
What Approaches and Strategies Were Used in the Analyses?
Item-Level Analysis (Layer 1)
Item-Level Analysis and Group-Level Characteristics (Layers 1, 5; Layers 1, 6)
Group of Items/Test-Level Analysis (Layer 2)
Group of Items/Test-Level Analysis and Personal Characteristics (Layers 2, 3)
Group of Items/Test-Level Analysis and Group-Level Characteristics (Layers 2, 3, 6)
Conclusion
The Potential and Limitations of Process Data in International Large-Scale Assessments
How Could the Ecological Framework Contribute to the Advancement of the Field?
What Is There to Come?
References
Part VIII: Methodology: Potential and Methods of Linking ILSA to National Education Policy and Research
34 Extending the ILSA Study Design to a Longitudinal Design
Introduction
Findings on the Impact of Tracking
Tracking in Czech Lower Secondary Education
Design of the Czech Longitudinal Study in Education
CLoSE Study Sampling and Data Collection
Measures of Achievement and Questionnaires
Mathematics Test
Czech Language
Reading Comprehension
Learning to Learn Competence Assessment
Questionnaires Used in Grades 5, 6, and 9
The Study of the Transition from Primary School to the Long Academic Track
Research Questions
Data and Methods
Results
Discussion of Findings on the Transition to the Long Academic Track
The Study of the Effects of the Long Academic Track on Students´ Achievement
Research Questions
Data and Methods
Results
Discussion of the Effects of the Long Academic Track on Student Progress
Conclusion/Summary: Pros and Cons of Using an Extension of TIMSS & PIRLS for National Purposes
References
35 Extending International Large-Scale Assessment Instruments to National Needs: Germany´s Approach to TIMSS and PIRLS
Introduction
Adaption of Survey Instruments
Example of Rewording to Ensure Cultural Appropriateness
Teacher Education (Teacher Questionnaire)
Extending Questionnaires
Example I: Describing Social Heterogeneity in Society by Looking at Relevant Subgroups (Extending the Home Questionnaire)
Example II: Addressing the Multidimensional Outcomes of Teaching and Learning (Extension of Student Questionnaire)
Example III: Investigating the Quality of Instruction (Extension of Student Questionnaire)
Example IV: Capturing Education Reform Programs (Extending the School Questionnaire)
Adapting or Extending Survey Documentation
Furthering Research: Perspectives that Will Answer Future Questions
Additional Longitudinal Component
Adding a Mixed-Method Component
Conclusion/Summary
References
36 Extending the Sample to Evaluate a National Bilingual Program
Introduction
Method
Participants
Instruments: Reading Comprehension Test
Predictor Variables of Academic Achievement
Procedure
Booklet Rotation Procedure
Data Analysis
Results
Conclusions
Appendix
References
37 A Non-Western Perspective on ILSAs
Introduction
Cultural and Educational Background
History of Education in the Gulf States
Societal Structure
Overview on the Educational Systems in the Gulf States
Results from Gulf States´ Participation in ILSAs
Participation in ILSAs
Results from ILSAs
Overall Performance on Primary and Secondary Level
Performance in Relation to Gender (TIMSS and PIRLS)
Performance in Relation to Nationality Status (TIMSS)
Educational Aspirations
Impact and Use of ILSAs in the Region
Summary of the TIMSS and PIRLS Encyclopedias
Policymaking
Curriculum
School Feedback
Teacher Training
Awareness and Student Motivation
Implementing Educational Reforms in Relation to Student Achievement
Showcase: Impact of TIMSS and PIRLS on the Omani Education System
Conclusion
Cross-References
References
Part IX: Findings: Schools, Principals, and Institutions
38 A Systematic Review of Studies Investigating the Relationships Between School Climate and Student Outcomes in TIMSS, PISA, ...
Introduction
Theoretical Framework
Multidimensionality of School Climate
The Aspects of School Climate
Academic
Community
Safety
Institutional Environment
School Climate and Student Outcomes
The Present Study
Methods
Search Procedures
Eligibility Criteria
Screening and Selection Process
Coding and Data Extraction
Findings
RQ1: How Is School Climate Assessed in ILSAs?
RQ2: What Characterizes the Studies Investigating the Relation Between School Climate Aspects and Student Outcomes Using ILSA ...
Aim of the Studies
Data and Samples
Methodological Appropriateness
RQ3: What Is the Pattern of Findings from the Studies Investigating the Relation Between School Climate Aspects and Student Ou...
Safety
Academic Climate
Community
Institutional Environment
Discussion and Implications
The Assessment of School Climate in ILSAs
A Systematic Review of ILSA-Related Research on the Relationships Between School Climate and Student Outcomes
Pattern of Findings from the Systematic of ILSAs Studies
Conclusion
Cross-References
References
Part X: Findings: Classrooms, Teachers, and Curricula
39 Teaching Quality and Student Outcomes in TIMSS and PISA
Introduction
Theoretical Background
ILSAs and Teaching Quality: History, Challenges, and Affordances
Brief Overview of the History of Teaching Quality in TIMSS and PISA
Challenges and Affordances of Using TIMSS and PISA for Measuring Teaching Quality
The TBD Framework
Teaching Quality in TIMSS: A Systematic Review
Methods
Search Procedure
Screening Process
Coding and Data Extraction
Results
Assessment of Teaching Quality in TIMSS (RQ1)
TIMSS Frameworks
Questionnaires
Characteristics of Studies Utilizing TIMSS (RQ2)
Samples
Outcomes
Methods
Measures of TQ
Patterns of Findings
Summary and Discussion of the Systematic Review
Teaching Quality in PISA
Getting Started: Teaching Quality in PISA 2000-2009
Defining and Implementing Measures of Teaching
Early Findings and Publications on Teaching Quality in PISA 2000-2009
PISA 2000 (Reading)
PISA 2003 (Mathematics)
PISA 2006 (Science)
PISA 2009 (Reading)
Implementing a Coherent Approach to the Measurement of Teaching Across Domains: PISA 2012, 2015, and 2018
Defining and Implementing an Overarching Conceptualization of Teaching
Cutting Across Domains: Integrating Data on Teaching Quality from PISA 2012, 2015, and 2018
Research on Teaching Mathematics Based on PISA 2012
Research on Science Teaching Based on PISA 2015
Validating and Checking Measurement Quality for PISA Scales on Teaching Quality
Reliability
Measurement Invariance
Validity
Discussion
Discussion of Results
Frameworks
Secondary Analyses Studies
Relations to Student Outcomes
Limitations, Contributions, and Concluding Remarks
References
40 Inquiry in Science Education
Introduction
Theoretical Framework
Inquiry as an Instructional Approach and Outcome
The Assessment of Inquiry in TIMSS
Inquiry as an Instructional Approach
Inquiry as an Instructional Outcome
The Assessment of Inquiry in PISA
Inquiry as an Instructional Approach
Inquiry as an Instructional Outcome
The Present Study
Methods
Literature Search
Inclusion Criteria
Search and Screening Process
Findings
RQ1: The Main Characteristics of the Studies
Aim of the Studies
TIMSS and/or PISA Data Analyzed in the Studies
The Measurement and Analysis of Inquiry
RQ2: Contribution of the Studies to the Research on Inquiry in Science Education
Inquiry as an Instructional Approach
Inquiry as an Instructional Outcome
Inquiry as an Instructional Approach and Outcome
Discussion
Conclusion/Summary
Cross-References
References
41 Teacher Competence and Professional Development
Introduction
Theoretical Background on Teacher Competence
Conceptualizing Teacher Competence
Conceptualization of GPK in Large-Scale Assessments
Teachers´ Situation-Specific Skills
Teacher Competence as a Predictor and an Outcome of Quality Education
Findings from ILSA on Teacher Competence
Measuring Teacher Competence in ILSA
Teacher Education
Affective-Motivational Facets of Teacher Competence
Teaching Experience
Professional Development
Associations between Teacher Characteristics and Student Learning Outcomes
Teacher Education
Affective-Motivational Facets of Teacher Competence
Professional Development
The Structure and Development of Teacher Competence with a Focus on Knowledge and Skills
Discussion
Summary
Outlook
References
42 Teachers´ Beliefs
Introduction
Purpose of This Chapter
Background on Teachers´ Professional Beliefs
Teachers´ Self-Efficacy
Teachers´ Satisfaction
Professionalism
Teachers´ Commitment
Leadership Beliefs
Data and Methods
Results
Research Proliferation: ILSA Studies on Teachers´ Professional Beliefs
Research Publications: Conceptualization of Teachers´ Professional Beliefs
Research Results: A Synthesis
Opportunities for Future Research
Conclusion
References
43 Homework: Facts and Fiction
Introduction
Homework Practices in Different Countries
Do Countries with More Homework Have Better Results?
Student Homework Behavior
Student Homework Time and Academic Achievement
Does the Time Spent on Homework Matter or Not?
The ``How´´ Is More Important Than the ``How Much´´
The Role of the Teachers
Family Support with Homework
Conclusions and Future Perspectives
Summary of Key Findings and Some Conclusions
Future Research Directions
References
Part XI: Findings: Students, Competences, and Dispositions
44 International Achievement in Mathematics, Science, and Reading
Overview
TIMSS 2015
PIRLS 2016
TIMSS 2015 and PIRLS 2016 International Benchmarks
Primary School Achievement in Mathematics
Fourth-Grade Mathematics: Achievement Descriptions at International Benchmarks
Fourth-Grade Students Reaching the Low and High TIMSS International Benchmarks in Mathematics
Primary School Achievement in Science
Fourth-Grade Science: Achievement Descriptions at International Benchmarks
Fourth-Grade Students Reaching the Low and High TIMSS International Benchmarks in Science
Primary School Achievement in Reading
Fourth-Grade Reading: Achievement Descriptions at International Benchmarks
Fourth-Grade Students Reaching the Low and High TIMSS International Benchmarks in Reading
Secondary School Achievement in Mathematics
Eighth-Grade Mathematics: Achievement Descriptions at International Benchmarks
Eighth-Grade Students Reaching the Low and High TIMSS International Benchmarks in Mathematics
Secondary School Achievement in Science
Eighth-Grade Science: Achievement Descriptions at International Benchmarks
Eighth-Grade Students Reaching the Low and High TIMSS International Benchmarks in Science
Conclusion/Summary
References
45 Digital Competences: Computer and Information Literacy and Computational Thinking
Introduction
Background
ICT as a Cross-Curricular Learning Area
Two Domains of Digital Competence
Historical Background of Assessing Digital Competence
Assessing Digital Competence in Cross-National Contexts
The International Computer and Information Literacy Study (ICILS)
Survey Development and Design
The Measurement of Computer and Information Literacy (CIL)
The Measurement of Computational Thinking (CT)
The Measurement of Contextual Variables
Results and Interpretations
Results from ICILS 2018 Regarding Students´ CIL and CT
The Context for Digital Learning
Explaining Variation in CIL and CT
Conclusion
Interpretations and Implications for Policy and Practice
References
46 Student Motivation and Self-Beliefs
Introduction
How Can Student Self-Beliefs and Motivation be Understood?
Assessment of Student Self-Beliefs and Motivation in ILSA Contexts
The ILSA Motivational Frameworks
Complex Constructs, Confusing Terminology?
Research on Motivation and Self-Beliefs Using ILSA Data
Are Motivation and Self-Beliefs Associated with Performance in ILSA Contexts?
Self-Beliefs and Motivation in Different Countries and Cultures: Can Findings be Validly Compared?
Other Group Differences in Levels of Motivation and Self-Beliefs
How Is ILSA Motivation Research Situated within the Larger Motivation Research Field?
Current Trends and Future Directions
Concluding Remarks
References
47 Well-Being in International Large-Scale Assessments
Introduction
What Is Child Well-being?
Criticism of Measuring Well-being in ILSAs
Including Well-being in ILSAs
Cognitive Dimension
Psychological Dimension
Social Dimension
Material: Economic Dimension
Physical Dimension
What Can Be Learnt from ILSAs About Children´s Well-being?
What Can Be Learnt from ILSAs About Adults´ Well-being?
Conclusions and Implications
References
Part XII: Findings: Equity and diversity
48 Gender Differences in School Achievement
Introduction
Studying Gender Differences in Achievement in ILSAs
Research Questions and Methods of Inquiry
Describing Gender Differences in ILSAs
Gender Gaps in Reading
Gender Gaps in Reading in Fourth-Grade Students
Gender Gaps in Reading in 15-Year-Old Students
Gender Gaps in Mathematics
Gender Gaps in Mathematics in Fourth-Grade Students
Gender Gaps in Mathematics in Eighth-Grade Students
Gender Gaps in Mathematics in 15-Year-Old Students
Gender Gaps in Science
Gender Gaps in Science in Fourth-Grade Students
Gender Gaps in Science in Eighth-Grade Students
Gender Gaps in Science in 15-Year-Old Students
Gender Gaps in Civic and Citizenship
Gender Gaps in Computer and Information Literacy
Secondary Analyses on Gender Differences in ILSAs
Reading Achievement
Explanatory Variables at the Student Level
Explanatory Variables at the Teacher and School Level
Explanatory Variables at the Country Level
Studies on Gender Gaps in the Variation of Reading Scores
Mathematics Achievement
Explanatory Variables at the Student Level
Explanatory Variables at the Teacher and School Level
Explanatory Variables at the Country Level
Studies on Gender Gaps in the Variation of Mathematics Scores
Association Between Reading and Mathematics Gender Gaps
Science Achievement
Explanatory Variables at the Individual and Family Level
Explanatory Variables at the Teacher and School Level
Explanatory Variables at the Country Level
Studies on Gender Gaps in the Variation of Science Scores
Civic and Citizenship Achievement
Computer and Information Literacy
Discussion
Overall Results
Methods of Analysis of the Secondary Studies
Implications of the Literature Review for Future Research
Embedding the Findings in a Broader Societal and Historical Context
Conclusion
Appendix: Details of the Literature Search
References
49 Dispersion of Student Achievement and Classroom Composition
Introduction
Learning Environment and the Student Body
Why Do Schools and Classrooms Differ in the Composition of Their Student Body?
Why Do We Assume that the Composition of the School or Classroom Affects Students´ Individual Development?
Operationalization of Student Composition
Overview of Findings on Achievement Composition
Descriptive Results on Achievement Dispersion
Effects of Achievement Dispersion and Level of a Group on Individual Student Achievement
Effects of Achievement Dispersion on Individual Student Achievement
Effects of Classroom or School Achievement Level on Individual Student Achievement
Differential Effects for High and Low Achievers
Summary of the Key Findings
Effects of Achievement Dispersion and Level of a Group on Individual Motivational and Psychosocial Outcomes of Students
Effects of Achievement Dispersion on Individual Motivational and Psychosocial Outcomes
Effects of Achievement Level on Individual Motivational and Psychosocial Outcomes
Differential Effects for High and Low Achievers
Summary of the Key Findings
Discussion
Limitations and Knowledge Gaps for Future Research
Criticisms of Research on Classroom and School Composition
Possible Ways of Addressing this Criticism
Future Directions
Composition of the Student Body: An Equity Perspective
Summary and Conclusion
References
50 Perspectives on Equity: Inputs Versus Outputs
Introduction
The Study of Equity and the Study of Equity with ILSAs
How to Address Equity: The Links Between Inputs and Outputs, and Between Equity and Excellence
An Analysis of Equity in Inputs and Outcomes for Students and Adults
Conclusions
Appendix A
Appendix B: Definition of Outcomes
PISA
PIAAC
References
51 Family Socioeconomic and Migration Background Mitigating Educational-Relevant Inequalities
Introduction
Data Collection Method
Keywords in the Literature Search
Selection Criteria and Screening Process
Measurement of Socioeconomic Status
Socioeconomic Status: Concept and Indicators Available
Operationalization of Socioeconomic Status in SES-Achievement Studies
The Function of Socioeconomic Status in Analyses
Analytical Methods Applied in the Studies
Effects of SES on Academic Achievement
SES Effect Based on Different Indicators
Effect of Cultural Capital on Individual Achievement
School Composition Effect on Student Outcome
Measurement of Family Migration
Migration Background: Concept and Indicators Available
Operationalization of Migration Background in SES-Achievement Studies
Functions of Migration Background
Effect of Migration on Achievement
Effect of Region of Origin
Effects of Migration at the School Level
Varying SES Effect on Migration Status
Summary of the Findings
Discussion Remarks
Appendix A
Literature Search Command in ProQuest
Search 1: SES and Migration = 101 Results
Search 2: Migration Only = 242 Results
Search 3: SES Only = 848 Results
Appendix B
Articles in the Final Round of Literature Review, Considered in the Chapter
Appendix C
Indicators of Family Socioeconomic Status in PISA, TIMSS, and PIRLS in each Cycle
References
Part XIII: Concluding Remarks
52 60-Years of ILSA: Where It Stands and How It Evolves
Meta-Perspectives on ILSAs in Education
Theoretical Meta-Perspectives: Educational Accountability and the Role of International Assessments
Reasons for Participation in ILSA
The Political Economy of ILSAs in Education: The Role of Knowledge Capital in Economic Growth
Educational Accountability and the Role of ILSAs in Economically Developed and Developing Countries
Theoretical Frameworks and Assessed Domains in ILSA
Comprehensive Frameworks in ILSA
Assessed Domains and Content Coverage
Populations and Design in ILSAs
Populations
Quantitative and Qualitative approaches in ILSA
Methods in ILSA
Generalizability and Comparability of Results
Analytical Potential of ILSA Data for Causal Analysis
Trend Analysis with ILSA Data
Log-Data
Findings
Schools, Principals, and Institutions
Classrooms, Teachers, and Curricula
Students, Competences, and Dispositions
Equity and Diversity
Final Remarks
References
Index
