The Domain Testing Workbook

CONTENTS......Page 7
A SCHEMA FOR DOMAIN TESTING: AN OVERVIEW ON ONE PAGE......Page 4
DEDICATION......Page 6
Testing Techniques......Page 19
Domain Testing is Not the Only Technique......Page 20
Testers......Page 21
Programmers?......Page 22
Section 3 Provides a Collection of Examples......Page 23
The Examples are Simple, Increasing in Complexity......Page 24
We Write in a Spiral......Page 25
Don’t Just Read the Examples—Work Through Them Yourself......Page 26
Books for Practitioners......Page 27
Why Write an Entire Book on Domain Testing?......Page 28
ACKNOWLEDGMENTS......Page 31
SECTION 1: WHAT IS DOMAIN TESTING?......Page 33
An Example Test Series......Page 34
The Example Test Series Again, Using the Schema......Page 37
1. Characterize the Variable......Page 38
2. Analyze the Variable and Create Tests......Page 40
3. Generalize to Multidimensional Variables......Page 49
Domains......Page 52
Equivalence Classes......Page 53
Selecting Representatives......Page 57
Automating Domain Tests......Page 60
Documenting and Exploring......Page 61
SECTION 1. PART 2. SUMMARIES OF KEY TECHNICAL CONCEPTS......Page 63
What’s Special About Black-Box Testing......Page 65
Sometimes You will be Stuck with Black-Box Testing......Page 66
BBST®......Page 68
Domain......Page 69
Specified Domain......Page 70
Variable......Page 71
Result Variable......Page 72
Output Variable......Page 74
Configuration Variable......Page 75
Stored Data......Page 76
User Interface Control......Page 77
Primary Versus Secondary Dimensions......Page 78
Why do You Need Notional Variables?......Page 79
Primary and Secondary Dimensions......Page 80
Explicit Dimensions......Page 81
Implicit Dimensions......Page 82
N-Tuples......Page 83
Combinatorial Explosion......Page 84
Risk......Page 86
Theory of Error......Page 87
Test Idea......Page 88
Error Revealing......Page 89
Corner Case......Page 90
Should All Tests be Powerful?......Page 91
Filters......Page 92
Testing Past the Filter......Page 93
An Analogy: How Domain Testing is Like Opinion Polling......Page 95
Stratified Sampling......Page 97
Partitioning Compared to Stratifying......Page 98
Linearizable Variables......Page 100
Structural Coverage......Page 101
There is More to Coverage than Code Coverage......Page 102
Coverage-Focused Test Techniques......Page 103
Try to Domain Test Every Variable......Page 104
SECTION 2: WORKING THROUGH THE SCHEMA’S TASKS......Page 105
SECTION 2. PART 1: CHARACTERIZE THE VARIABLE......Page 107
From Example 3: SunTrust VISA......Page 111
From Example 27: OpenOffice Impress Grid Options......Page 112
Record-Keeping......Page 114
Exercises: Part 1-A......Page 115
From Example 6: Tax Table......Page 120
From Example 23: Frequent Flyer Miles......Page 121
Exercises: Part 1-B......Page 122
From Example 3: SunTrust VISA......Page 126
From Example 27: OpenOffice Impress Grid Options......Page 127
Exercises: Part 1-C......Page 128
Integers......Page 134
Fixed Point......Page 135
Floating Point......Page 136
The Relevance of Floating Point Math to Domain Testing......Page 138
Different Types of Floating-Point Numbers......Page 139
Tolerance Levels, Delta and Machine Epsilon......Page 141
A Few More Details About Floating Point......Page 142
Char......Page 144
String......Page 145
Record......Page 146
List......Page 147
Ordinal Scale......Page 148
Interval Scale......Page 149
From Examples 3, 5 and 8......Page 150
From Example 25: Moodle Assign User Roles......Page 151
Exercises: Part 1-D......Page 152
Character-Based Encoding......Page 156
How Big......Page 157
Timing......Page 158
Variation Between Things that Should be Equivalent......Page 159
Lead to the Same Consequence......Page 160
From Example 26: Install Printer Software......Page 161
Some Theorists Reserve Domain Testing for Interval and Ratio Variables......Page 162
From Example 10: The Circle......Page 164
From Example 24: Notepad Open File Dialog......Page 165
Exercises: Part 1-E......Page 166
F. DETERMINE WHETHER THIS IS AN INPUT VARIABLE OR A RESULT......Page 170
From Example 4: Page Setup Options......Page 171
Exercises: Part 1-F......Page 172
G. DETERMINE HOW THE PROGRAM USES THIS VARIABLE......Page 177
From Example 4: Page Setup Options......Page 178
From Example 23: Frequent-Flyer Miles......Page 180
Exercises: Part 1-G......Page 181
How Can you Identify Potentially-Related Variables?......Page 186
Constraint Relationships......Page 188
Joint Effect......Page 189
Risk in Common......Page 190
Competition for a Common Resource......Page 191
From Example 10: The Circle......Page 192
From Example 26: Install Printer Software......Page 193
Exercises: Part 1-H......Page 194
Practical Considerations in Working With Secondary Dimensions......Page 196
Prioritizing Your Tests......Page 197
Using Standard Lists for Secondary Dimensions......Page 198
I. PARTITION THE VARIABLE......Page 202
From Example 6: Tax Table......Page 203
From Example 28: OpenOffice Printing Options......Page 204
Analyzing Non-Ordered Dimensions......Page 205
From Example 26: Install Printer Software......Page 206
From Example 24: Notepad Open File Dialog......Page 207
Exercises: Part 2-1......Page 209
From Example 3: SunTrust VISA......Page 213
The Risk/Equivalence Table......Page 214
From Example 6: Tax Table......Page 215
The Risk/Equivalence Table......Page 216
From Example 8: Create Table (Columns, Rows)......Page 217
From Example 9: Create Table (Max Cells)......Page 218
Exercises: Part 2-J......Page 220
K. CREATE TESTS FOR THE CONSEQUENCES OF THE DATA ENTERED......Page 224
Calculate the Amount to Pay an Employee......Page 226
Add Columns to a Spreadsheet......Page 227
Solving a System of Linear Equations......Page 228
From Example 15: Mailing Labels......Page 230
Exercises: Part 2-K......Page 231
Secondary Dimensions......Page 236
From Example 1: Integer Input......Page 238
From Example 3: SunTrust VISA......Page 239
The Classical Boundary/Equivalence Table......Page 241
From Example 7: Student Names......Page 243
Exercises: Part 2-L......Page 245
M. SUMMARIZE YOUR ANALYSIS WITH A RISK/EQUIVALENCE TABLE......Page 249
Secondary Dimensions......Page 250
From Example 1: Integer Input......Page 251
Secondary Dimensions for Integers......Page 252
The Risk/Equivalence Table (Integers)......Page 255
From Example 3: SunTrust VISA......Page 257
Secondary Dimensions for Fixed-Point Variables......Page 258
The Risk/Equivalence Table (Fixed Point)......Page 261
Secondary Dimensions for Floating-Point Variables......Page 262
A Risk/Equivalence Table for a Floating-Point Variable......Page 265
From Example 7: Student Names......Page 269
Secondary Dimensions for Strings......Page 270
The Risk/Equivalence Table (Strings)......Page 272
Exercises: Part 2-M......Page 276
SECTION 2: PART 3: GENERALIZE TO MULTIDIMENSIONAL VARIABLES......Page 277
N-Tuple Notation......Page 279
Input Variables......Page 280
Internal Variables......Page 281
Testing m+n+p+q Variables Together......Page 282
Designing Combination Tests......Page 284
Independent Variables......Page 285
Non-Independent Variables......Page 286
(B) What Values of those Variables Should You Test?......Page 287
(C) How Should You Combine Values of Variables into Tests?......Page 288
(D) Coverage Criteria......Page 289
N. ANALYZE INDEPENDENT VARIABLES THAT SHOULD BE TESTED TOGETHER......Page 292
Testing Independent Variables......Page 293
Random Combinations......Page 294
All Singles......Page 295
All N-Tuples......Page 296
All Pairs, All Triples, etc.......Page 298
From Example 30: Configuration Testing......Page 301
Exercises: Part 3-N......Page 303
Equivalent Results......Page 306
Data Flows......Page 307
From Example 14: Result of Calculations......Page 308
From Example 17: Joan’s Pay......Page 311
Analysis of the xi’s.......Page 315
Exercises: Part 3-O......Page 316
Ways Variables Constrain Each Other......Page 320
Example 8: Create Table (Columns, Rows) (Variables that Multiply to a Maximum)......Page 322
Example 12: Piecewise Boundary (Linear Relationships)......Page 323
Example 10: The Circle (Nonlinear Relationships)......Page 326
(a) The Circle {(x1, x2) | x12 + x22 = 100}......Page 327
(b) Inside the Circle {(x1, x2) | x12 + x22 < 100}......Page 328
(d) The Circle & Points Inside it {(x1, x2) | x12 + x22 ≤ 100}......Page 329
Example 3: SunTrust VISA (Complex Relationships)......Page 330
Sometimes Related, Sometimes Not......Page 331
Exercises: Part 3-P......Page 332
SECTION 2. PART 4: PREPARE FOR ADDITIONAL TESTING......Page 334
Q. IDENTIFY AND LIST UNANALYZED VARIABLES. Gather Information for Later Analysis......Page 337
From Example 24: Notepad Open File Dialog......Page 339
Exercises: Part 4-Q......Page 340
R. IMAGINE AND DOCUMENT RISKS THAT DON’T MAP TO AN OBVIOUS DIMENSION......Page 345
From Example 3: SunTrust VISA......Page 346
From Example 25: Moodle Assign User Roles......Page 347
Exercises: Part 4-R......Page 349
SECTION 3: WORKED EXAMPLES......Page 352
SECTION 3. PART 1: EXAMPLES THAT ILLUSTRATE THE SCHEMA......Page 354
Example 1: Integer Input......Page 355
Test Idea Catalog for Integers......Page 357
Generic Risk/Equivalence Table for Integers......Page 360
Example 2: ATM Withdrawal......Page 364
Example 3: SunTrust VISA......Page 374
Test Idea Catalog for Fixed-Point Variables......Page 380
Generic Risk/Equivalence Table for Fixed-Point Variables......Page 383
Example 4: Page Setup Options......Page 386
Test Idea Catalog for Floating-Point Variables......Page 389
Secondary Dimensions in the Classical Table......Page 392
A Risk/Equivalence Table for a Floating-Point Variable......Page 393
Example 5: Undergraduate Grades......Page 401
Example 6: Tax Table......Page 406
Showing Multiple Valid Equivalence Classes on the Same Table......Page 408
Example 7: Student Names......Page 413
Test Idea Catalog for Strings......Page 417
Generic Risk/Equivalence Table for Strings......Page 420
Example 8: Create Table (Columns, Rows)......Page 425
Example 9: Create Table (Max Cells)......Page 431
Example 10: The Circle......Page 437
Choosing On Points and Off Points for Testing Nonlinear Relationships......Page 441
(a) The Circle {(x1, x2) | x12 + x22 = 100}......Page 443
(b) Inside the Circle {(x1, x2) | x12 + x22 < 100}......Page 445
(d) The Circle & Points Inside it {(x1, x2) | x12 + x22 ≤ 100}......Page 446
(e) The Circle & Points Outside it {(x1, x2) | x12 + x22 ≥ 100}......Page 447
Example 11: University Admissions......Page 448
Tests Based on a Black-Box Model may Differ from Tests Based on the Code......Page 452
Risk/Equivalence Table Showing Multiple Related Variables......Page 453
Analysis on the Assumption the Implementation is Equivalent to Case B......Page 454
Example 12: Piecewise Boundary......Page 458
Choosing On Points and Off Points for Testing Linear Relationships......Page 461
Additional Notes......Page 463
Example 13: Sum of Two Integers......Page 464
Example 14: Result Of Calculations......Page 469
Example 15: Mailing Labels......Page 476
Constraint Satisfaction Problems......Page 485
Why Should YOU Care about Geometric Boundary Problems?......Page 487
What Should You Do with Problems like This?......Page 488
Example 16: Unemployment Insurance Benefits......Page 492
Example 17: Joan’s Pay......Page 498
Example 18: Volume Discounts......Page 505
Example 19: Sum Of Squares......Page 510
Analysis of Example 20......Page 518
Example 21: Spreadsheet Sort Table......Page 521
Case 1: One-Column, Homogeneous Orderable Data......Page 527
Cases 4, 5 and 6: Multiple Columns......Page 529
Cases 7,8 and 9: Sort by Multiple Columns......Page 530
Example 22: NextDate......Page 537
Example 23: Frequent-Flyer Miles......Page 543
Example 24: Notepad Open File Dialog......Page 549
A Test Idea Catalog for File Name Strings......Page 552
Grouping File Names by Type......Page 566
SECTION 3. PART 2: ADVANCED EXAMPLES......Page 569
Overview......Page 571
Where Can You Apply Domain Analysis?......Page 574
Overview......Page 578
Where Can You Apply Domain Analysis?......Page 579
Overview......Page 583
Where Can You Apply Domain Analysis?......Page 586
Where Can You Apply Domain Analysis?......Page 588
Where Can You Apply Domain Analysis?......Page 590
Overview......Page 592
Where Can You Apply Domain Analysis?......Page 593
A Schema for Domain Testing......Page 594
Using Domain Testing when You are Just Learning the Program......Page 596
Simple Domain Testing has Limits and Will Run Out of Steam......Page 597
As You Learn More about the Program, Use Domain Testing in a Deeper Way......Page 599
What Makes These Tests Domain Tests?......Page 600
Combine Domain Testing with Other Techniques to Increase Their Efficiency and Power......Page 601
University Courses on Software Testing......Page 604
Activity-Focused Design......Page 605
Coverage-Focused Design......Page 606
Backward Design......Page 607
We Recommend Assessment-Focused Design......Page 609
Levels of Knowledge......Page 610
Appropriate Evaluation......Page 611
Appropriate Instruction......Page 612
Transfer of Learning and the Evolution of Our Schema......Page 613
For more information......Page 617
REFERENCES......Page 618
The Classical Boundary/Equivalence Table......Page 630
The Risk/Equivalence Table......Page 631