The Handbook of Institutional Research

The Handbook of Institutional Research......Page 5
Copyright......Page 6
Contents......Page 9
List of Tables......Page 13
List of Figures......Page 15
List of Exhibits......Page 17
Appendices......Page 19
Preface......Page 21
Overview and Organization of the Handbook......Page 22
Handbook Development......Page 23
References......Page 24
Acknowledgments......Page 25
About the Editors......Page 27
List of Contributors......Page 29
The Association for Institutional Research......Page 33
Part 1: The History, Theory, and Practice of Institutional Research......Page 37
College Self-Study......Page 39
Surveys......Page 40
Research Committees/Bureaus......Page 41
Advocacy for Establishment of Institutional Research Offices......Page 42
National Institutional Research Forums......Page 44
The Association for Institutional Research......Page 45
Establishment of AIR Affiliated Regional, State, and Special Interest Groups......Page 46
Alternative Views of IR: Research, Administrative Support, or Self-Study?......Page 47
Interpreting and Providing Examples of IR to IR Practitioners......Page 48
Interpreting IR to the Broader Higher Education Community......Page 50
Adding Perspective: Some Key Sources......Page 51
Concluding Note......Page 53
References......Page 54
2: The Structure and Functions of Institutional Research Offices......Page 58
The Organizational Characteristics of Institutional Research......Page 59
The Evolution of IR......Page 60
An Institutional Research Ecology......Page 63
Measuring the Maturity of Institutional Research......Page 67
Hierarchy of IR Tasks......Page 68
The Characteristics of a Mature Profession: Does IR Enjoy These?......Page 70
The Faces of Institutional Research......Page 72
References......Page 74
Standards of Professional Practice and Codes of Ethics in Institutional Research......Page 76
Codes of Ethics......Page 77
Description, Synthesis, and Assessment of Skills Needed in Institutional Research......Page 79
Issues Intelligence......Page 80
Contextual Intelligence......Page 81
An Assessment of the Profession's Skills and Intelligences......Page 83
Acquiring Skills for Effective Institutional Research......Page 86
Skills' Relation to Institution Types and IR Office Locations......Page 89
Conclusion......Page 90
References......Page 91
4: The Role of Institutional Research in International Universities......Page 93
World-Class Universities......Page 94
Universities Outside the United States......Page 96
Regional Cooperation to Improve Higher Education......Page 97
U.S. Institutions Functioning Abroad......Page 101
International Universities Based on the U.S. Model......Page 103
Conclusion......Page 105
References......Page 106
5: Out of the Box and Out of the Office: Institutional Research for Changing Times......Page 109
Context for Innovation and Leadership......Page 110
Individual Context......Page 111
Organizational Skills......Page 112
Cultural Skills......Page 113
Interpersonal Skills and Leadership......Page 115
Assessment......Page 116
Strategic Planning......Page 117
Accreditation......Page 118
Curriculum Development......Page 119
Synthesis......Page 120
References......Page 121
The Uses and Limits of IR as a Decision-Support Activity......Page 122
Defining OL and Understanding a Collaborative Approach......Page 125
Concepts to Facilitate Collaborative Organizational Learning......Page 127
Argyris and Schön’s Single- and Double-Loop Learning......Page 128
Huber's Taxonomy for Organizational Learning......Page 129
Situated Learning and Communities of Practice......Page 135
Implications for IR Products, Processes, and Services......Page 136
Implications for the Skills and Dispositions Useful to IR......Page 138
Notes......Page 139
References......Page 140
The Expanding Role of IR Units: The New Frontier of Managing Change......Page 143
Motivation: Philosophy, Social-Ecological Systems, and the Pressures of Change......Page 144
Change Process Theories, Models, and Types......Page 146
The Range of Change Types: From Simple to Complex......Page 147
Exciting Change by Encouraging Doubt: Placing Data in High-Use Areas......Page 149
Barriers to Change: How IR/A Units Can Mitigate Their Impact......Page 150
IR/A Role in Building Institutional Resiliency and Adaptive Capacity......Page 152
The CSI Change Management Process......Page 153
Adaptive Management: Evoking Change by Creating Platforms for Provoking Doubt......Page 154
Institutionalizing Change Management: Strengthening Institutional Resiliency......Page 156
Evidence of System Impact on Academic Culture......Page 160
Evidence of Change Activity and Success: An Index That Monitors Change Characteristics......Page 161
Notes......Page 163
References......Page 164
Part 2: Supporting Campus Leadership and Management......Page 167
8: Supporting Institutional Governance......Page 169
Special Relationship......Page 170
Presidential Realities......Page 171
Types of Executive Requests......Page 174
What Is Shared and with Whom?......Page 176
IR Support of Shared Governance......Page 177
Conclusion......Page 179
References......Page 180
Context for Supporting the Provost......Page 181
The Importance of IR in Supporting Academic Institutional Effectiveness......Page 182
Keys to Providing Timely and Distilled Information to the Provost......Page 183
Relevance to the Mission of the Institution......Page 184
The Balancing Act of Complex Interactions......Page 185
Efficiency Versus Effectiveness......Page 187
Academic Support: Teaching and Research......Page 188
IR Support of Teaching......Page 189
IR Support of Research......Page 195
References......Page 198
Evaluating Faculty Recruitment......Page 201
Data from Academic Units......Page 203
Data from Recruited Faculty Members......Page 204
Analyzing Faculty Retention......Page 205
Data from Academic Units......Page 206
Data from Faculty Members......Page 207
Analyzing Promotion and Tenure Decisions......Page 210
Projecting Retirements......Page 212
Projecting Number of Hires, Salary Outlays, and Faculty Mix......Page 214
Assessing Faculty Job Satisfaction......Page 215
Strengths and Weaknesses......Page 216
References......Page 217
Institutional Research, Planning, and Budgeting......Page 219
Conceptions of Strategic Management......Page 220
Multiple Institutions, Multiple Contexts......Page 221
Setting Priorities, Making Choices......Page 222
Strategic Review of Academic Programs......Page 223
A Role for Institutional Research......Page 224
The Review and Recommendation Process......Page 225
College and University Budgeting......Page 226
It's Still a People Problem......Page 227
Create Strong Resource Allocation Tools Before They Are Needed......Page 228
Planning to Plan......Page 229
Import-Export Model: The Orienting Role of the Environment......Page 230
Focusing on a Limited Number of Goals......Page 233
Integrated Planning......Page 234
Implications and Conclusion......Page 235
Plan Implementation......Page 236
References......Page 237
12: Building Cost Models......Page 239
The Basics: Types of Cost Models and Types of Cost......Page 241
Focus on Cost Accounting Procedures to Determine Instructional Costs......Page 243
Factors and Drivers in Cost Models......Page 245
Cost Models for Technology......Page 247
Building One Type of Instructional Cost Model......Page 248
References......Page 253
A Common Theme......Page 257
Recruiting Students......Page 258
Admitting Students......Page 259
Awarding Financial Aid......Page 260
Placing Students......Page 262
Retaining and Graduating Students......Page 263
Differences by Institutional Type......Page 265
New Forms of Educational Delivery......Page 266
Targeting Resources for Maximum Effect......Page 267
Ubiquitous Data......Page 268
College Rankings and Accountability Metrics......Page 269
The Future of Enrollment Management......Page 270
References......Page 271
14: Refocusing Student Success: Toward a Comprehensive Model......Page 273
The Federal Role in Shaping Current Metrics of Student Success......Page 274
Limitations of the Traditional Graduation Rate Measure in Today's Environment......Page 275
Progress in Data System Development Facilitating Improved Graduation Rate Measures......Page 276
Promising Initiatives to Expand or Improve on Existing Graduation Rate Measures......Page 278
Expanding and Refocusing What Constitutes Student Success......Page 279
University of Alaska Anchorage (UAA) Model Overview......Page 280
Data Design and Operational Methodology......Page 281
Model Design or Implementation Challenges......Page 286
Model Status To Date......Page 287
Beyond UAA and the Future......Page 288
References......Page 289
Appendix 14.1: UAA Decision Rules for Student Status Determination......Page 290
15: Academic Space Management and the Role of Institutional Research......Page 292
Why Does Space Matter?......Page 293
Moving from Traditional Systems to Databases......Page 294
The Metrics of Academic Space Management......Page 297
The Role of Institutional Research in Space Management......Page 300
References......Page 303
16: Managing Sustainability......Page 304
Understanding the Context......Page 305
Understanding the Tone at the Top......Page 307
Auditing and Tracking the Context......Page 308
Higher Education Auditing Frameworks......Page 309
Certification Initiatives......Page 316
External Uses of Sustainability Data......Page 318
Preparing for Participation......Page 320
References......Page 323
Appendix 16.1: The Talloires Declaration......Page 326
Appendix 16.2: Calculating a STARS Score......Page 328
Appendix 16.3: Survey Questions for the Princeton Review Green Rating of Colleges......Page 329
Part 3: Bridging Internal and External Requirements for IR......Page 331
17: Challenges in Meeting Demands for Accountability......Page 335
Increasing Demands for Accountability......Page 336
To Whom?......Page 338
For What?......Page 339
Internal Accountability......Page 340
Federal-Level Accountability......Page 341
Federal Institutional Disclosure and Reporting Requirements......Page 342
Federal Program Disclosure and Reporting Requirements......Page 343
References......Page 344
A Brief History of Accreditation......Page 346
Institutional Research and Accreditation......Page 347
Institutional Effectiveness and the Expectations of Regional Accreditors......Page 348
Institutional Research and Regional Accreditation......Page 349
State Accreditation......Page 351
Specialized or Programmatic Accreditation......Page 353
Reviewing the Curriculum and Providing Evidence of Student Learning......Page 354
Evaluating a Variety of Resources......Page 356
Accreditation, Accountability, and the Educational Landscape......Page 357
References......Page 358
Appendix 18.1: Accrediting Agencies' Websites......Page 360
19: Regulated Ethics: Institutional Research Compliance with IRBs and FERPA......Page 361
Standards for Review......Page 363
When Should IR Projects Be Submitted for IRB Review?......Page 366
Exempt, Expedited, or Full Board Review—Who Makes the Call?......Page 367
Projects Contracted with Outside Entities......Page 368
Assessment......Page 369
Family Education Records Privacy Act......Page 371
State and Local Reporting......Page 372
Conclusion......Page 373
References......Page 374
Appendix 19.1: FERPA Concerns to Address in Agreements with Third Parties for Disclosure of Identifiable Data for Educational Research......Page 375
Our Environment and Culture......Page 376
The Role of Statistics......Page 377
Background......Page 378
Title VII of the Civil Rights Act of 1964......Page 379
What Is Fair Versus What Is Legal......Page 381
Statistics and the Courts......Page 382
Federal Rules of Evidence......Page 383
A Difference in Establishing What Is Truth......Page 384
The Nature of Research......Page 385
Conclusions Differ for Different Cases in Different Courts......Page 386
Conclusion......Page 387
References......Page 388
U.S. Department of Education......Page 390
NCES Datasets......Page 391
The Integrated Postsecondary Data System......Page 392
IPEDS Data Tools......Page 393
Collaboration with Institutional Research Community......Page 396
Surveys......Page 397
NSF Data Access......Page 399
Conclusions......Page 400
References......Page 401
Appendix 21.1: NCES Sample Surveys......Page 402
Appendix 21.2: IPEDS Surveys......Page 403
Appendix 21.3: NSF Surveys......Page 404
22: Collective Responses to a New Era of Accountability in Higher Education......Page 407
Overview of Initiatives......Page 408
University and College Accountability Network (U-CAN)......Page 409
Voluntary System of Accountability (VSA)......Page 411
Transparency by Design (TbD)......Page 415
Voluntary Framework of Accountability (VFA)......Page 417
References......Page 419
Appendix 22.1: Initiative Websites, Resources, and Reports......Page 421
The System and State Environment......Page 422
Role of Governing and Coordinating Boards......Page 423
Institutional Research for Governing and Coordinating Boards......Page 425
Strategic Planning......Page 427
Stewardship and Accountability......Page 429
Special Research and Policy Studies......Page 434
Preferences of Decision Makers......Page 436
References......Page 437
Evolution of K–20+ Education in the United States......Page 440
K–20+ Data Systems......Page 441
Elements of K–20+ Databases: An Overview......Page 442
Design and Implementation: Data In......Page 446
Unique Identifier......Page 447
Database Design......Page 448
Standard Reports......Page 449
Student-Level Data......Page 450
Current Context......Page 451
Conclusion......Page 453
References......Page 454
25: Data Exchange Consortia: Characteristics, Current Examples, and Developing a New Exchange......Page 456
Membership Criterion: Invitational Versus Mandatory......Page 457
Examples of Data Exchange Consortia......Page 458
Consortium for Student Retention Data Exchange......Page 459
National Community College Benchmark Project......Page 460
Polytechnic Data Sharing for Higher Educational Research......Page 461
University 15 Data Exchange......Page 462
Creating a Data Exchange......Page 463
Problem Identification Phase......Page 464
Development Phase......Page 465
Review Phase......Page 468
References......Page 469
26: Business Intelligence and Analytics: The IR Vision for Data Administration, Reporting, Data Marts, and Data Warehousing......Page 470
Business Intelligence......Page 471
Analytics......Page 472
Data Administration......Page 473
Managing the Data......Page 475
Design and Efficiency......Page 476
Access/Security......Page 477
Extract, Transform, Load......Page 478
Data Integrity......Page 479
Reporting, Topical Data Applications, and Data Marts......Page 480
Data Warehousing......Page 481
Business Intelligence and Analytics in the Context of Institutional Research......Page 484
References......Page 486
Part 4: Institutional Research Tools and Techniques......Page 491
27: Analytic Approaches to Creating Planning and Decision Support Information......Page 495
Types of Measurement......Page 496
Using Analytics......Page 497
Specific Types of Techniques......Page 498
Data Sources......Page 505
Analysis......Page 506
Mixed Methods Approaches and Methods......Page 507
Resources and the Role of Assumptions......Page 508
Assumptions......Page 509
References......Page 510
The Scope and Nature of Exploratory Methods......Page 514
Differences Between Hypothesis Testing and Exploratory Methods......Page 515
Cluster Analysis......Page 517
Cluster Analysis Example: Student Engagement Typologies......Page 518
Clustering with K-Means Method......Page 519
Applications for Institutional Research......Page 522
Decision Trees......Page 523
Data Selection and Exploration......Page 524
Model Building......Page 527
Artificial Neural Networks......Page 529
Multilayer Perceptron Architecture......Page 530
Example: Using Neural Networks as a Part of Overall Analysis......Page 532
Applications for Institutional Research......Page 534
Recommendation 3: Adopt algorithmic bias analysis......Page 535
References......Page 536
Step 1: Choose a Method......Page 538
Step 3: Develop Research Questions......Page 539
Surveys and Questionnaires......Page 540
Designing a Survey Study......Page 541
Interpreting Survey Data......Page 548
Focus Groups in Institutional Research Settings......Page 550
Focus Group Sampling......Page 551
Developing Focus Group Moderators......Page 552
Conducting the Focus Group......Page 553
Analyzing Focus Group Data......Page 554
Interpretation of the Data......Page 555
Observation of Established Institutional Practices and Procedures......Page 556
References......Page 557
Published Instruments: Overview of Cognitive Tests......Page 559
Undergraduate Admission Tests......Page 560
Course Placement Tests......Page 562
Undergraduate Honors Placement and Credit by Exam......Page 566
Published Instruments: Overview of Noncognitive Measures......Page 567
Recruitment......Page 570
Admission......Page 572
Course Placement......Page 574
Identifying At-Risk Students......Page 576
Other Uses of Published Instruments in Institutional Research......Page 577
Advantages and Limitations of Using Published Instruments in IR......Page 578
Concerns in Using Data from Published Instruments......Page 579
Role of the Institutional Researcher......Page 581
References......Page 582
31: Measuring and Evaluating Faculty Workload......Page 586
Current Data Sources......Page 587
The National Study of Instructional Costs and Productivity......Page 590
The Kansas Study of Community College Instructional Costs and Productivity......Page 595
The National Community College Benchmark Project......Page 598
Selected Measures of Out-of-Classroom Faculty Activity Study......Page 601
Conclusions......Page 606
References......Page 607
32: Analyzing Equity in Faculty Compensation......Page 609
Purposes of Salary Equity Studies......Page 610
Current Issues in Salary Studies......Page 622
Faculty Productivity......Page 623
Legal Considerations......Page 624
Salary Compression......Page 625
Conclusions......Page 626
References......Page 627
Preparing for Effective Reporting......Page 630
The Research Checklist......Page 631
What Other Information Gets Conveyed in the Reporting Process?......Page 632
Access to Information......Page 633
What Goes Into a Report?......Page 634
Types of Reports......Page 635
Components of Reports......Page 636
The Role of the Presenter and the Presentation Software......Page 637
Designing Effective Tables......Page 638
Presenting Quantitative Data in Graphs......Page 639
Principles of Effective Graphics......Page 640
Other Rules of Presentation......Page 641
Reporting Qualitative Data......Page 642
Distributing Information and Communicating Results......Page 643
The Role of the Internet......Page 644
References......Page 645
34: Tools for Setting Strategy......Page 647
PESTEL Analysis......Page 648
SWOT......Page 650
Portfolio Analysis Using the Ansoff Matrix......Page 654
Benchmarks and Benchmarking......Page 655
Benchmarking Process......Page 656
Conclusion......Page 658
References......Page 660
Key Performance Indicators......Page 661
Categories of KPIs: A Balance of Perspectives......Page 662
Changing or Updating KPIs......Page 664
IR Roles and Process for Developing KPIs......Page 665
Balanced Scorecards and Dashboards......Page 666
Users of Balanced Scorecards in Higher Education Institutions......Page 667
Scorecard or Dashboard Display and Presentation: The Role of Technology......Page 669
Strategy Execution: An Emerging Role for Balanced Scorecards and Dashboards......Page 670
References......Page 679
36: Developing Institutional Comparisons......Page 680
Data Exchanges and Consortia......Page 681
Targeted Data Collections......Page 682
IPEDS Data Feedback Report......Page 683
Types of Groups......Page 684
Methods for Developing Groups......Page 686
Decision Support......Page 689
References......Page 690
37: Tools for Improving Institutional Effectiveness......Page 692
Business Theory: Misunderstandings and Cultural Differences......Page 693
The Organization as a Collection of Relationships and Processes......Page 694
The Construct of an Institutional Culture......Page 695
The Closed System of Maintaining an Effective Organization......Page 697
Specific Tools for Identifying Where to Begin......Page 698
Specific Tools for Looking at the Process......Page 699
Other Process Management Analytic Tools......Page 702
Conclusion......Page 705
References......Page 706
Appendix 37.1: Resources......Page 708
How Do We Define "Effectiveness" In IR?......Page 709
The IR Self-Study......Page 711
Workflow......Page 713
Cost-Effectiveness......Page 714
Getting Customer Feedback......Page 715
Seeking External Feedback......Page 717
Volunteer Peer Audits......Page 718
Process Benchmarking......Page 719
Performance Benchmarking......Page 720
References......Page 721
It Is Not Over Until It Is Over......Page 725
Sometimes You Are the Windshield, Sometimes You Are the Bug......Page 726
Denial, Hostility, Bargaining, Depression, and Acceptance......Page 727
Quality Is a Verb, and Change Is a Journey......Page 728
References......Page 729
Name Index......Page 731
Subject Index......Page 743