Software Testing: A Craftsman’s Approach

Cover
Half Title
Title Page
Copyright Page
Dedication
Table of Contents
Preface
Authors
Part I: A Mathematical Context
Chapter 1: A Perspective on Testing
1.1 Basic Definitions
1.2 Test Cases
1.3 Insights from a Venn Diagram
1.4 Identifying Test Cases
1.4.1 Specification-based Testing
1.4.2 Code-based Testing
1.4.3 The Specification-based versus Code-based Debate
1.5 Fault Taxonomies
1.6 Levels of Testing
Exercises
References
Chapter 2: Examples
2.1 Structural Elements of Pseudo-code and Java
2.2 The Triangle Problem
2.2.1 Problem Statement
2.2.2 Discussion
2.2.3 Java Implementation
2.3 The NextDate Function
2.3.1 Problem Statement
2.3.2 Discussion
2.3.3 Java Implementation
2.4 The Foodies-Wish-List Online Shopping Application
2.4.1 Problem Statement
2.4.2 Discussion
2.5 The Garage Door Controller
2.6 Examples in Exercises
2.6.1 The Quadrilateral Program
2.6.2 The NextWeek Function
2.6.3 The Windshield Wiper Controller
Exercises
References
Chapter 3: Discrete Math for Testers
3.1 Set Theory
3.1.1 Set Membership
3.1.2 Set Definition
3.1.3 The Empty Set
3.1.4 Venn Diagrams
3.1.5 Set Operations
3.1.6 Set Relations
3.1.7 Set Partitions
3.1.8 Set Identities
3.2 Functions
3.2.1 Domain and Range
3.2.2 Function Types
3.2.3 Function Composition
3.3 Relations
3.3.1 Relations among Sets
3.3.2 Relations on a Single Set
3.4 Propositional Logic
3.4.1 Logical Operators
3.4.2 Logical Expressions
3.4.3 Logical Equivalence
3.4.4 Probability Theory
Exercises
Reference
Chapter 4: Graph Theory for Testers
4.1 Graphs
4.1.1 Degree of a Node
4.1.2 Incidence Matrices
4.1.3 Adjacency Matrices
4.1.4 Paths
4.1.5 Connectedness
4.1.6 Condensation Graphs
4.1.7 Cyclomatic Number
4.2 Directed Graphs
4.2.1 Indegrees and Outdegrees
4.2.2 Types of Nodes
4.2.3 Adjacency Matrix of a Directed Graph
4.2.4 Paths and Semipaths
4.2.5 Reachability Matrix
4.2.6 n-Connectedness
4.2.7 Strong Components
4.3 Graphs for Testing
4.3.1 Program Graphs
4.3.2 Finite State Machines
4.3.3 Petri Nets
4.3.4 Event-Driven Petri Nets
4.3.5 Statecharts
Exercises
Reference
Part II: Unit Testing
Chapter 5: Boundary Value Testing
5.1 Normal Boundary Value Testing
5.1.1 Generalizing Boundary Value Analysis
5.1.2 Limitations of Boundary Value Analysis
5.2 Robust Boundary Value Testing
5.3 Worst Case Boundary Value Testing
5.4 Special Value Testing
5.5 Examples
5.5.1 Test Cases for the Triangle Problem
5.5.2 Test Cases for the NextDate Function
5.6 Random Testing
5.7 Guidelines for Boundary Value Testing
Exercises
Chapter 6: Equivalence Class Testing
6.1 Equivalence Classes
6.2 Traditional Equivalence Class Testing
6.3 Improved Equivalence Class Testing
6.3.1 Weak Normal Equivalence Class Testing
6.3.2 Strong Normal Equivalence Class Testing
6.3.3 Weak Robust Equivalence Class Testing
6.3.4 Strong Robust Equivalence Class Testing
6.4 Equivalence Class Test Cases for the Triangle Problem
6.5 Equivalence Class Test Cases for the NextDate Function
6.6 Equivalence Class Test Cases for the completeOrder Method
6.7 “Edge Testing”
6.8 Reflections on Invalid Classes
6.9 Guidelines and Observations
Exercises
References
Chapter 7: Decision Table-Based Testing
7.1 Decision Tables
7.2 Decision Table Techniques
7.3 Test Cases for the Triangle Problem
7.4 Test Cases for the NextDate Function
7.4.1 First Try
7.4.2 Second Try
7.4.3 Third Try
7.5 Cause and Effect Graphing
7.6 Guidelines and Observations
Exercises
References
Chapter 8: Code-Based Testing
8.1 Program Graphs
8.2 DD-Paths
8.3 Code Coverage Metrics
8.3.1 Program Graph-Based Coverage Metrics
8.3.2 E. F. Miller’s Coverage Metrics
8.3.2.1 Statement Testing
8.3.2.2 DD-Path Testing
8.3.2.3 Simple Loop Coverage
8.3.2.4 Predicate Outcome Testing
8.3.2.5 Dependent Pairs of DD-Paths
8.3.2.6 Complex Loop Coverage
8.3.2.7 Multiple Condition Coverage
8.3.2.8 “Statistically Significant” Coverage
8.3.2.9 All Possible Paths Coverage
8.3.3 A Closer Look at Compound Conditions
8.3.3.1 Boolean Expression (per Chilenski)
8.3.3.2 Condition (per Chilenski)
8.3.3.3 Coupled Conditions (per Chilenski)
8.3.3.4 Masking Conditions (per Chilenski)
8.3.3.5 Modified Condition Decision Coverage
8.3.4 Examples
8.3.4.1 Condition with Two Simple Conditions
8.3.4.2 Example: Compound Condition from NextDate
8.3.4.2.1 Program Graph-Based Coverage Metrics
8.3.4.2.2 Model-Based (Decision Table) Code Coverage Metrics
8.3.4.2.3 Compound Condition from the Triangle Program
8.3.4.3 Test Coverage Analyzers
8.3.4.4 Java Code for Tests in Table 8.8
8.3.4.5 Junit Test Results
8.3.4.6 Capabilities of Selected Code Coverage Tools
8.4 Basis Path Testing
8.4.1 McCabe’s Basis Path Method
8.4.2 Observations on McCabe’s Basis Path Method
8.4.3 Essential Complexity
8.5 Guidelines and Observations
Exercises
References
Chapter 9: Testing Object-Oriented Software
9.1 Unit Testing Frameworks
9.1.1 Common Unit Testing Frameworks
9.1.2 JUnit Examples
9.2 Mock Objects and Automated Object Mocking
9.3 Dataflow Testing
9.3.1 Define/Use Testing Definition
9.3.2 Define/Use Testing Metrics
9.3.3 Define/Use Testing Example
9.4 Object-Oriented Complexity Metrics
9.4.1 WMC—Weighted Methods per Class
9.4.2 DIT—Depth of Inheritance Tree
9.4.3 NOC—Number of Child Classes
9.4.4 CBO—Coupling Between Classes
9.4.5 RFC—Response for Class
9.4.6 LCOM—Lack of Cohesion on Methods
9.5 Issues in Testing Object-Oriented Software
9.5.1 Implications of Composition and Encapsulation
9.5.2 Implications of Inheritance
9.5.3 Implications of Polymorphism
9.6 Slice-Based Testing
9.6.1 Example
9.6.2 Style and Technique
9.6.3 Slice Splicing
9.6.4 Program Slicing Tools
Exercises
References
Chapter 10: Retrospective on Unit Testing
10.1 The Test Method Pendulum
10.2 Traversing the Pendulum
10.2.1 Program Graph-Based Testing
10.2.2 Basis Path Testing
10.2.3 Dataflow Testing
10.2.4 Slice-Based Testing
10.2.5 Boundary Value Testing
10.2.6 Equivalence Class Testing
10.2.7 Decision Table Testing
10.3 Insurance Premium Case Study
10.4 Specification-Based Testing
10.4.1 Code-Based Testing
10.4.1.1 Path-based Testing
10.4.1.2 Dataflow Testing
10.4.1.3 Slice Testing
10.5 Guidelines
Exercises
References
Part III: Beyond Unit Testing
Chapter 11: Life Cycle-Based Testing
11.1 Traditional Waterfall Testing
11.1.1 Waterfall Testing
11.1.2 Pros and Cons of the Waterfall Model
11.2 Testing in Iterative Lifecycles
11.2.1 Waterfall Spin-Offs
11.2.2 Specification-Based Life Cycle Models
11.3 Agile Testing
11.3.1 About User Stories
11.3.1.1 Behavior-Driven Development
Rules 3 and 4
Rules 5 and 6
11.3.1.2 Use Cases
11.3.2 Extreme Programming
11.3.3 Scrum
11.3.4 Test-Driven Development
11.3.5 Agile Model-Driven Development
11.3.6 Model-Driven Agile Development
11.4 Remaining Questions
11.4.1 Specification or Code Based?
11.4.2 Configuration Management?
11.4.3 Granularity?
11.5 Pros, cons, and Open Questions of TDD
11.6 Retrospective on MDD vs. TDD
References
Chapter 12: Integration Testing
12.1 Decomposition-Based Integration
12.1.1 Top-down Integration
12.1.2 Bottom-up Integration
12.1.3 Sandwich Integration
12.1.4 Pros and Cons
12.2 Call Graph-Based Integration
12.2.1 Pairwise Integration
12.2.2 Neighborhood Integration
12.2.3 Pros and Cons
12.3 Path-Based Integration
12.3.1 New and Extended Concepts
12.3.2 MM-Path Complexity
12.3.3 Pros and Cons
12.4 Example: Procedural integrationNextDate
12.4.1 Decomposition-Based Integration
12.4.2 Call Graph-Based Integration
12.4.3 Integration Based on MM-Paths
12.4.4 Observations and Recommendations
12.5 Example: O-O integrationNextDate
12.6 Model-Based Integration Testing
12.6.1 Message Communication
12.6.2 Pairwise Integration
12.6.3 FSM/M Path Integration
12.6.4 Scenario 1: Normal Account Creation
Exercises
References
Chapter 13: System Testing
13.1 Threads
13.1.1 Thread Possibilities
13.1.2 Thread Definitions
13.2 Identifying Threads in Single-Processor Applications
13.2.1 User Stories/Use Cases
13.2.2 How Many Use Cases?
13.2.2.1 Incidence with Input Events and Messages
13.2.2.2 Incidence with Output Actions and Messages
13.2.2.3 Incidence with Classes
13.2.3 Threads in Finite State Machines
13.2.3.1 Paths in a Finite State Machine
13.2.3.2 How Many Paths?
13.2.4 Atomic System Functions
13.3 Identifying Threads in Systems of Systems
13.3.1 Dialogues
13.3.2 Communicating FSMs
13.3.3 Dialogues as Sequences of ASFs
13.4 System Level Test Cases
13.4.1 An Industrial Test Execution System
13.4.2 Use Cases to Test Cases
13.4.3 Finite State Machine Paths to Test Cases
13.4.4 Dialogue Scenarios to Test Cases
13.4.5 Communicating Finite State Machines to Test Cases
13.5 Coverage Metrics for System Testing
13.5.1 Use Case-Based Test Coverage
13.5.2 Model-Based Test Coverage
13.6 Long Versus Short Test Cases
13.6.1 Supplemental Approaches to System Testing
13.6.2 Operational Profiles
13.6.2.1 Risk-Based Testing
13.7 Non-functional System Testing
13.7.1 Stress Testing Strategies
13.7.1.1 Compression
13.7.1.2 Replication
13.7.2 Mathematical Approaches
13.7.2.1 Queueing Theory
13.7.2.2 Reliability Models
13.7.2.3 Monte Carlo Testing
Exercises
References
Chapter 14: Model-Based Testing
14.1 Testing Based on Models
14.2 Appropriate Models
14.2.1 Peterson’s Lattice
14.2.2 Expressive Capabilities of Mainline Models
14.2.3 Modeling Issues
14.2.4 Making Appropriate Choices
14.3 Commercial Tool Support for Model-Based Testing
14.3.1 TestOptimal
14.3.2 Conformiq
14.3.3 Verified Systems International GmbH
Exercises
References
Chapter 15: Software Complexity
15.1 Unit Level Complexity
15.1.1 Cyclomatic Complexity
15.1.1.1 “Cattle Pens” and Cyclomatic Complexity
15.1.1.2 Node Outdegrees and Cyclomatic Complexity
15.1.1.3 Decisional Complexity
15.1.2 Computational Complexity
15.1.2.1 Halstead’s Metrics
15.1.2.2 Example: Day of Week with Zeller’s Congruence
15.2 Integration Level Complexity
15.2.1 Integration Level Cyclomatic Complexity
15.2.2 Message Traffic Complexity
15.3 Software Complexity Example
15.4 Object-Oriented Complexity
15.4.1 WMC—Weighted Methods per Class
15.4.2 DIT—Depth of Inheritance Tree
15.4.3 NOC—Number of Child Classes
15.4.4 CBO—Coupling between Classes
15.4.5 RFC—Response for Class
15.4.6 LCOM—Lack of Cohesion on Methods
15.5 System Level Complexity
15.5.1 Cyclomatic Complexity of Source Code
15.5.2 Complexity of Specification Models
15.5.3 Use Case Complexity
15.5.4 UML Complexity
Exercise
References
Chapter 16: Testing Systems of Systems
16.1 Characteristics of Systems of Systems
16.2 Sample Systems of Systems
16.2.1 The Garage Door Controller (Directed)
16.2.2 Air Traffic Management System (Acknowledged)
16.2.3 The Foodie Wish List System
16.3 Software Engineering for Systems of Systems
16.3.1 Requirements Elicitation
16.3.2 Specification with a Dialect of UML
16.3.2.1 Air Traffic Management System Classes
16.3.2.2 Air Traffic Management System Use Cases and Sequence Diagrams
Normal Landing Use Case
November 1993 Incident Use Case
16.3.3 Testing
16.4 Communication Primitives for Systems of Systems
16.4.1 ESML Prompts as Petri Nets
16.4.1.1 Petri Net Conflict
16.4.1.2 Petri Net Interlock
16.4.1.3 Enable, Disable, and Activate
16.4.1.4 Trigger
16.4.1.5 Suspend and Resume
16.4.2 New Prompts as Swim Lane Petri Nets
16.4.2.1 Request
16.4.2.2 Accept
16.4.2.3 Reject
16.4.2.4 Postpone
16.4.2.5 Swim Lane Description of the November 1993 Incident ( Figure 16.19)
16.5 Effect of Systems of Systems Levels on Prompts
16.5.1 Directed and Acknowledged Systems of Systems
16.5.2 Collaborative and Virtual Systems of Systems
Exercises
References
Chapter 17: Feature Interaction Testing
17.1 Feature Interaction Problem Defined
17.2 Types of Feature Interactions
17.2.1 Input Conflict
17.2.2 Output Conflict
17.2.3 Resource Conflict
17.3 A Taxonomy of Interactions
17.3.1 Static Interactions in a Single Processor
17.3.2 Static Interactions in Multiple Processors
17.3.3 Dynamic Interactions in a Single Processor
17.3.4 Dynamic Interactions in Multiple Processors
17.4 Interaction, Composition, and Determinism
Exercises
References
Chapter 18: Case Study:: Testing Event-Driven Systems
18.1 The Garage Door Controller Problem Statement
18.2 Modeling with Behavior Driven Development (BDD)
18.3 Modeling with Extended Finite State Machines
18.3.1 Deriving a Finite State Machine from BDD Scenarios
18.3.2 Top-down development of a Finite State Machine
18.4 Modeling with Swim Lane Event-Driven Petri Nets
18.4.1 Normal Garage Door Closing
18.4.2 Garage Door Closing with an Intermediate Stop
18.4.3 Garage Door Closing with a Laser Beam Crossing
18.4.4 The Door Opening Interactions
18.5 Deriving Test Cases from Swim Lane Event-Driven Petri Nets
18.6 Failure Mode Event Analysis (FMEA)
Exercises
References
Chapter 19: A Closer Look at All Pairs Testing
19.1 The All Pairs Technique
19.1.1 Program Inputs
19.1.2 Independent Variables
19.1.3 Input Order
19.1.4 Failures Due only to Pairs of Inputs
19.2 A Closer Look at the NIST Study
19.3 Appropriate Applications for All-Pairs Testing
19.4 Recommendations for All Pairs Testing
Exercises
References
Chapter 20: Software Technical Reviews
20.1 Economics of Software Reviews
20.2 Types of Reviews
20.2.1 Walkthroughs
20.2.2 Technical Inspections
20.2.3 Audits
20.2.4 Comparison of Review Types
20.3 Roles in a Review
20.3.1 Producer
20.3.2 Review Leader
20.3.3 Recorder
20.3.4 Reviewer
20.3.5 Role Duplication
20.4 Contents of an Inspection Packet
20.4.1 Work Product Requirements
20.4.2 Frozen Work Product
20.4.3 Standards and Checklists
20.4.4 Review Issues Spreadsheet
20.4.5 Review Reporting Forms
20.4.6 Fault Severity Levels
20.4.7 Review Report Outline
20.5 An Industrial-Strength Inspection Process
20.5.1 Commitment Planning
20.5.2 Reviewer Introduction
20.5.3 Preparation
20.5.4 Review Meeting
20.5.5 Report Preparation
20.5.6 Disposition
20.6 Effective Review Culture
20.6.1 Etiquette
20.6.2 Management Participation in Review Meetings
20.6.3 A Tale of Two Reviews
20.6.3.1 A Pointy-Haired Supervisor Review
20.6.3.2 An Ideal Review
20.7 Inspection Case Study
References
Chapter 21: Epilogue: Software Testing Excellence
21.1 Craftsmanship
21.2 Best Practices of Software Testing
21.3 Our Top 10 Best Practices for Software Testing Excellence
21.3.1 Carefully Performed Technical Inspections
21.3.2 Careful Definition and Identification of Levels of Testing
21.3.3 Model-Based Testing at All Levels
21.3.4 System Testing Extensions
21.3.5 Incidence Matrices to Guide Regression Testing
21.3.6 Use of xUnit and Object Mocking at the Unit Level
21.3.7 Intelligent Combination of Specification-Based and Code-Based Unit Level Testing
21.3.8 Use of Appropriate Tools at All Testing Levels
21.3.9 Exploratory Testing During Maintenance
21.3.10 Test-Driven Development
21.4 Mapping Best Practices to Diverse Projects
21.4.1 A Mission Critical Project
21.4.2 A Time Critical Project
21.4.3 Corrective Maintenance of Legacy code
21.5 An Extreme Example
References
Appendix A: Complete Technical Inspection Packet
A.1 Customer Requirements: ATM Simulator
A.2 Base Use Cases
A.3 Base Use Case Standard
A.4 Base Use Case Checklist
A.5 Base Use Case Fault Severity Levels
A.6 Base Use Case Technical Inspection Forms
A.7 Sample Inspection Report Outline
Table of Contents
Attachments
Appendix B: Foodies Wish List Example
B.1 General Description
B.2 Messages Among Finite State Machines
B.2.1 Foodie Wish List Finite State Machines
B.2.1.1 Foodie Home
B.2.1.2 Account Creation
B.2.1.3 Login
B.2.1.4 Shopping List
B.2.1.5 Admin
B.2.1.6 Shopping Cart
B.2.1.7 FoodieDB
B.3 Dialogs Across Swim Lanes
B.3.1 Scenarios
B.3.1.1 Scenario 1.1: Normal Account Creation
B.3.1.2 Scenario 1.2: Duplicate UserID found
B.3.1.3 Scenario 1.3: Partial Account Creation (no PIN definition)
B.3.1.4 Scenarios and Test Coverage for Login
B.3.1.4.1 Scenario 2.1: Valid Login, PIN correct on 1 st try
B.3.1.4.2 Scenario 2.2: Valid Login, PIN correct on 2nd try
B.3.1.4.3 Scenario 2.3: Valid Login, PIN correct on 3rd try
B.3.1.4.4 Scenario 2.4: Invalid Login, PIN failed on 3rd try
B.3.1.4.5 Scenario 2.5: Invalid Login, no PIN try
B.3.1.5 Summary of Event/Message Sequences for Scenarios 2.1 to 2.5
B.3.1.6 Test Coverage of Scenarios 2.1–2.5
Scenario 4: End-to-End Login to Purchase Transaction
B.4 Object-Oriented Design
Index
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Y
Z