This book is about the design and development of tools for software testing. It intends to get the reader involved in software testing rather than simply memorizing the concepts. The source codes are downloadable from the book website. The book has three parts: software testability, fault localization, and test data generation.
The ability to test software efficiently and thoroughly is a hallmark of high-quality software. For code to be tested thoroughly, it should be testable in the first place. Testers, in a sense, are customers of developers. Developers must undertake an approach that keeps the code clean and efficient. Clean and efficient code is highly testable. Refactoring techniques help develop testable code. As the testability improves, automatic test data generation tools can provide relatively more efficient and effective test data. This book offers a machine learning tool to measure software testability based on which a software development process, Testability-Driven Development (TsDD), in support of test-first approaches is suggested.
Testing is something practical. Software testing is all practice. It relies on knowledge but is more than learning. It is based on programming experience and common sense. This book is all about the design and development of tools for software testing. My intention is to get the reader involved in software testing rather than simply memorizing the concepts. The book has three parts: Software Testability, Fault Localization, and Test data generation.
Part I describes unit and acceptance tests and proposes a new method called testability-driven development (TsDD) in support of TDD and BDD. TsDD uses a machine learning model to measure testability before and after refactoring. The reader will learn how to develop the testability prediction model and write software tools for automatic refactoring.
Part II focuses on developing tools for automatic fault localization. This part shows the reader how to use a compiler generator to instrument source code, create control flow graphs, identify prime paths, and slice the source code. On top of these tools, a software tool, Diagnoser, is offered to facilitate experimenting with and developing new fault localization algorithms. Diagnoser takes a source code and its test suite as input and reports the coverage provided by the test cases and the suspiciousness score for each statement.
Part III proposes using software testing as a prominent part of the cyber-physical system software to uncover and model unknown physical behaviors and the underlying physical rules. The reader will get insights into developing software tools to generate white box test data.
I have intentionally begun Chap. 1 of this book with the concept of testability. A straightforward yet fundamental concept overlooked by many programmers. As a software quality factor, testability is the extent to which a software artifact supports its testing. This chapter brings up some critical issues about software preparation for testing.
Chapter 2 briefly describes using Enterprise Architect to define business and technical requirements and generate code from UML models. The chapter proceeds with the practical use of test-driven development (TDD) to complete the generated code. This chapter provides guidelines and step-by-step instructions for installing and using Nunit in Visual Studio and Unitest in Python. In this chapter, the reader will learn how to use requirements specifications and UML models to derive test cases and how to apply TDD.
Chapter 3 describes behavior-driven development (BDD) as an agile methodology to design and develop code based on the acceptance tests satisfying the system requirements defined as user stories. This chapter gives the reader insights into the strengths and weaknesses of different tools support for different programming languages, Java, C#, and Python. BDD supports acceptance testing, while TDD is based on unit tests that test the individual methods in isolation.
Author(s): Saeed Parsa
Publisher: Springer
Year: 2023
Language: English
Pages: 594
Preface
Acknowledgments
Source codes
About This Book
Contents
Part I Testability Driven Development
1 Software Testability
1.1 Introduction
1.2 Testability
1.3 Clean Code
1.3.1 Clean Code Principles
1.3.2 Roslyn Compiler
1.3.3 Using Roslyn to Clean Code
1.4 Testability Metrics
1.4.1 Object-Oriented System Metrics
1.4.2 Observability and Controllability
1.4.3 Dependency Metrics
1.5 Modularity and Testability
1.5.1 Modularity Principles
1.5.2 Automatic Re-modularization
1.6 Design Patterns
1.6.1 Design Patterns and Testability
1.7 Summary
1.8 Exercises
References
2 Unit Testing and Test-Driven Development (TDD)
2.1 Introduction
2.2 Unit Testing
2.2.1 Generating a Test Class with NUnit
2.2.2 Reading Test Cases from a File
2.2.3 Testing in Isolation
2.3 Test-Driven Development (TDD)
2.3.1 TDD Life Cycle for Object-Oriented Software
2.3.2 Single Responsibility Principle
2.4 Unit Testing in Python
2.4.1 Nose
2.4.2 Unit Test
2.4.3 PyTest
2.5 Summary
2.6 Exercises
References
3 Acceptance Testing and Behavior Driven Development (BDD)
3.1 Introduction
3.2 Acceptance Testing
3.3 User Stories
3.3.1 Acceptance Criteria
3.3.2 Specification by Example
3.3.3 Acceptance Criteria/Tests
3.3.4 Background Section
3.3.5 Example Table
3.3.6 Gherkin Language
3.4 Behavior-Driven Development (BDD)
3.4.1 Case Study: A Medical Prescription Software System
3.4.2 Defining Goals
3.4.3 Defining Features
3.4.4 Executable Specifications
3.4.5 Low-Level Specification
3.4.6 Creating Test Functions
3.4.7 Application Code
3.4.8 Behave Reports
3.4.9 Behave Hooks
3.5 BDD Process
3.5.1 Example
3.5.2 Defining Features
3.5.3 Executable Specifications
3.5.4 Low-Level Specification
3.5.5 Hooks
3.6 Using the Behave Software Tool for BDD in Python
3.6.1 Installing Behave
3.6.2 Tutorial
3.6.3 A Telemedicine System
3.6.4 Acceptance Tests
3.6.5 Step Definition
3.6.6 Step Implementation
3.6.7 Unit Testing in Python
3.6.8 Project File Structure
3.6.9 Report Generation Using Allure
3.7 Using SpecFlow for BDD
3.7.1 Installation
3.7.2 Create SpecFlow Project
3.7.3 Create Feature Files
3.7.4 Create Executable Specifications
3.7.5 Low-Level Specifications
3.7.6 Code Implementation
3.7.7 Acceptance and Unit Tests
3.8 Summary
3.9 Exercises
References
4 Testability Driven Development (TsDD)
4.1 Introduction
4.2 Testability-Driven Development
4.2.1 TDD Elaboration
4.2.2 Testability-Driven Paradigm
4.3 TsDD in Practice
4.3.1 Motivating Example-1
4.3.2 Experimental Results
4.4 Testability Mathematical Model
4.4.1 Test Effectiveness
4.4.2 Test Efficiency
4.4.3 Testability
4.4.4 Coverageability
4.5 Testability Prediction Model
4.5.1 Feature Selection
4.5.2 Dataset Preparation
4.5.3 Learning Models
4.5.4 Important Testability Metrics
4.5.5 Why Predicts Testability?
4.5.6 Summary
4.6 Assignment
References
5 Automatic Refactoring
5.1 Introduction
5.2 Encapsulate Field Refactoring
5.3 Automatic Refactoring
5.3.1 The ANTLR Parser Generator
5.3.2 Installing the ANTLR Parser Generator
5.3.3 Preparing the Grammar
5.4 Implementing Encapsulate Field Refactoring
5.4.1 The Main Function
5.4.2 Using the Listener Class
5.4.3 Encapsulate Field Refactoring Listener
5.4.4 Overriding the Listener Methods
5.5 Rename Method Refactoring
5.5.1 Meaningful Method Names
5.5.2 Method Vectorization
5.5.3 Cross-Project Naming
5.5.4 K-Nearest Neighbor Similarity
5.6 Refactorings
5.7 Extract Class
5.7.1 Class Cohesion
5.7.2 Extracting Fields and Methods
5.8 Move Method
5.8.1 Move Method Refactoring Implementation
5.9 Summary
5.10 Exercises
References
Part II Automatic Fault Localization
6 Graph Coverage
6.1 Introduction
6.2 Control Flow Graphs
6.2.1 Basic Control Flow Graphs
6.3 Graph Coverage Criteria
6.3.1 Statement Coverage
6.3.2 Branch Coverage
6.3.3 Decision Coverage
6.3.4 Clause Coverage
6.3.5 Path Coverage
6.3.6 Prime Paths
6.3.7 Basis Paths
6.4 Control Flow Graph Extraction
6.4.1 The Main Body of the CFG Extractor Program
6.4.2 CFG Extractor Visitor Class
6.4.3 Labeling Rule Alternatives
6.4.4 Function Definitions
6.4.5 Example
6.4.6 Compound Statements
6.4.7 Selection Statements
6.4.8 Iteration Statements
6.5 Control Flow Graphs for Python Functions
6.5.1 What is an AST?
6.5.2 Using AST Class in Python
6.5.3 Automatic Construction of CFGs for Python Programs
6.5.4 The Visitor Class
6.5.5 If Statements
6.5.6 Iteration Statements
6.6 Exercises
References
7 Spectrum-Based Fault Localization
7.1 Introduction
7.2 Spectrum-Based Fault Localization in Practice
7.2.1 A Motivating Example
7.2.2 Fault Localization
7.3 Source Code Instrumentation
7.3.1 Automatic Instrumentation
7.4 Summary
7.5 Exercises
References
8 Fault Localization Tools: Diagnosis Matrix and Slicing
8.1 Introduction
8.2 Diagnosis Matrix
8.3 Slicing
8.3.1 Backward Slicing
8.3.2 Forward Slicing
8.4 Program Dependency Graphs
8.4.1 Slicing Algorithm
8.5 Data Flow Analysis
8.5.1 Static Data Flow Testing
8.5.2 Dynamic Data Flow Testing
8.5.3 Reaching Definitions
8.5.4 Definition-Use Chains
8.6 Control Dependency Graphs
8.6.1 Dominator and Post-Dominant Trees
8.6.2 Control Dependency Graph
8.7 Summary
8.8 Exercises
References
9 Coincidentally Correct Executions
9.1 Introduction
9.2 Coincidental Correctness
9.2.1 Impact of CC Tests on SBFL (A Motivating Example)
9.3 Detecting Coincidentally Correct Executions
9.3.1 Computing the Probability of N-grams
9.3.2 Computing Cross-Entropies to Measure the Disorders
9.3.3 Detecting Coincidentally Correct Executions
9.3.4 Locating Suspicious N-grams
9.4 Writing a Program to Locate Faults
9.4.1 Source Code Instrumentation
9.4.2 Computing N-grams
9.4.3 Compute N-gram Probabiliies
9.4.4 Computing Chained Probabilities
9.4.5 Compute Cross-Entropy
9.4.6 Cluster Test Cases
9.4.7 Locating Suspicious N-grams
9.4.8 Computing Suspiciousness Scores
9.5 Summary
9.6 Exercises
References
10 Graph and Frequency-Based Fault Localization
10.1 Introduction
10.2 Graph-Based Fault Localization
10.2.1 A Running Example
10.3 Execution Graphs
10.4 Vector Similarity Measurement
10.4.1 Euclidian and Cosine Distances
10.4.2 Jaccard and Cosine Distances
10.5 Selecting the Most Similar Failing and Passing Path
10.6 Selecting the Most Suspicious Transitions (Branches)
10.7 Summary
10.8 Exercises
References
Part III Automatic Test Data Generation
11 Search-Based Testing
11.1 Introduction
11.2 Encoding
11.3 Test Data Generation Using Genetic Algorithms
11.3.1 Source Code Instrumentation
11.3.2 Branch Distance
11.3.3 Main Function
11.3.4 Initial Population
11.3.5 Termination Condition
11.3.6 Genetic Operators
11.3.7 Fitness Functions
11.3.8 Execution Results
11.4 MCMC Search Method
11.4.1 MCMC Sampling Method
11.4.2 Metropolis Hasting Algorithm
11.4.3 Examples
11.5 Importance of Domain Coverage
11.6 Exercises
References
12 Testing Unknowns
12.1 Introduction
12.2 Behavior Coverage
12.3 Rule Extraction for Autonomous Driving
12.3.1 Configuring the CARLA Simulator
12.3.2 CARLA Components
12.3.3 Extracting I/O Intensive Execution Paths
12.3.4 Detecting Unexpected Behavior
12.3.5 Fitness Evaluation and Ranking
12.4 Autopilot Software System Behavior
12.4.1 Monitoring Component
12.4.2 Test Results
12.4.3 Test Data Generation
12.4.4 Domain Generator
12.4.5 Fitness Function
12.4.6 Instrumenting the Simulator
12.4.7 Calculating Boundary Points
12.5 Summary
References
13 Automatic Test Data Generation Symbolic and Concolic Executions
13.1 Introduction
13.2 Dynamic Symbolic Execution
13.3 Symbolic Execution
13.3.1 Symbolic Execution Tree
13.3.2 Path Constraint
13.3.3 Path Explosion
13.3.4 Black Box Functions
13.4 Concolic Execution
13.4.1 Instrumentation for Concolic Execution
13.4.2 Solving the Path Explosion Problem
13.5 Basis Paths
13.6 Search Strategies
13.6.1 Classical Search Strategies
13.6.2 Heuristic Search Strategies
13.6.3 Integration of Search Strategies
13.6.4 Combined Search Strategies
13.6.5 A Function to ADD Nodes to an Execution Tree
13.6.6 A Function to Look for a Node in Execution Trees
13.7 Tools and Techniques
13.7.1 Crown
13.7.2 Microsoft Testing Tools
13.7.3 Parameterized Unit Testing
13.8 Summary
13.9 Exercises
References
14 Automatic Test Data Generation for Domain Coverage
14.1 Introduction
14.2 Random Testing
14.2.1 Path Random Testing
14.2.2 The IP-PRT Algorithm
14.3 Euler/Venn Reasoning for Test Data Generation
14.3.1 Euler/Venn Representation of Input Domains
14.3.2 DAG Implementation of Euler/Venn Diagrams
14.4 Domain Coverage Algorithm
14.4.1 Example
14.5 The Domain-Coverage Algorithm
14.6 A Simple Constraint Propagation Algorithm
14.6.1 Constraint Satisfaction
14.7 Interval Arithmetic
14.7.1 Dynamic Domain Reduction
14.7.2 Update Function
14.8 Why Domain Coverage
14.9 Summary
14.10 Exercises
References
