Performance Testing Guidance for Web Applications

Cover Page......Page 1
Title......Page 2
Copyrights Page......Page 3
Foreword By Alberto Savoia......Page 4
Foreword By Rico Mariani......Page 6
Scope of This Guide......Page 7
Features of This Guide......Page 8
Parts......Page 9
Approach Used in This Guide......Page 10
External Contributors and Reviewers......Page 12
Tell Us About Your Success......Page 13
Part I: Introduction to Performance Testing......Page 14
How to Use This Chapter......Page 15
Core Activities of Performance Testing......Page 16
Why Do Performance Testing?......Page 17
Project Context......Page 18
Tuning Process Overview......Page 20
Baselines......Page 21
Terminology......Page 23
Summary......Page 27
Performance Testing......Page 28
Key Types of Performance Testing......Page 29
Summary Matrix of Benefits by Key Performance Test Types......Page 30
Additional Concepts / Terms......Page 33
Summary......Page 34
Summary Matrix of Risks Addressed by Performance Testing Types......Page 35
Speed-Related Risks......Page 37
Scalability-Related Risks......Page 38
Stability-Related Risks......Page 39
Summary......Page 40
Part II: Exemplar Performance Testing Approaches......Page 41
Overview of Activities......Page 42
Summary Table of Core Performance-Testing Activities......Page 43
Core Performance-Testing Activities Walkthrough......Page 45
Activity 1. Identify the Test Environment......Page 46
Activity 2. Identify Performance Acceptance Criteria......Page 47
Activity 3. Plan and Design Tests......Page 48
Considerations......Page 49
Activity 5. Implement the Test Design......Page 50
Activity 6. Execute the Test......Page 51
Considerations......Page 52
Activity 7. Analyze Results, Report, and Retest......Page 53
Summary......Page 55
How to Use This Chapter......Page 56
Iterative Performance Testing Activities......Page 57
Activity 1. Understand the Project Vision and Context......Page 59
Activity 2. Identify Reasons for Testing Performance......Page 60
Activity 3. Identify the Value Performance Testing Adds to the Project......Page 61
Activity 4. Configure the Test Environment......Page 62
Activity 5. Identify and Coordinate Tasks......Page 63
Checklist......Page 64
Checklist......Page 65
Checklist......Page 66
Checklist......Page 67
Summary......Page 68
How to Use This Chapter......Page 69
Agile Performance-Testing Activities......Page 70
Relationship to Core Performance-Testing Activities......Page 72
Understand the System......Page 73
Understand the Performance Build Schedule......Page 74
Success Criteria......Page 75
Discussion Points......Page 76
Activity 5. Identify and Coordinate Tasks......Page 77
Activity 6. Execute Task(s)......Page 78
Activity 8. Revisit Activities 1-3 and Consider Performance Acceptance Criteria......Page 79
Additional Considerations......Page 80
Summary......Page 81
How to Use This Chapter......Page 82
CMMI Performance-Testing Activities......Page 83
Relationship to Core Performance-Testing Activities......Page 85
CMMI Performance Testing Activity Flow......Page 86
Determine the Process......Page 87
Understand the System......Page 88
Performance Goals......Page 89
Map Work Items to Project Plan......Page 90
Prioritize Work Items by Phase......Page 91
Add Detail to the Plan......Page 92
Determine Navigation Paths for Key Scenarios......Page 93
Determine the Relative Distribution of Scenarios......Page 94
Identify Metrics to Be Captured During Test Execution......Page 95
Activity 6. Configure the Test Environment......Page 96
Activity 9. Report Results and Archive Data......Page 97
Activity 12. Prepare the Final Report......Page 98
Summary......Page 99
Part III: Identify the Test Environment......Page 100
Approach for Evaluating the System......Page 101
Considerations......Page 102
Considerations......Page 103
Logical Architecture......Page 104
Physical Architecture......Page 106
Considerations......Page 107
Summary......Page 108
Part IV: Identify Performance Acceptance Criteria......Page 109
How to Use This Chapter......Page 110
Terminology......Page 111
Review the Project Plan......Page 112
Capture or Estimate Resource Usage Targets and Thresholds......Page 113
Capture or Estimate Resource Budgets......Page 114
Communicate Results......Page 115
Performance Objectives......Page 116
Performance Objectives......Page 117
Questions......Page 118
Questions......Page 119
Summary......Page 120
How to Use This Chapter......Page 121
Determine Application Functionality and Usage......Page 122
Review Project Documentation and Related Contracts......Page 123
Interview Stakeholders Who Will Influence the “Go Live” Decision......Page 124
Separate Requirements from Goals......Page 125
Quantify Captured Performance Goals......Page 126
Quantify Captured Performance Requirements......Page 127
Summary......Page 128
How to Use This Chapter......Page 129
Terminology......Page 130
Investigate End-User Requirements......Page 131
Collect Business Requirements......Page 132
Research Standards, Compliance, and Contracts......Page 133
Compare and Consolidate Performance Characteristics......Page 134
Summary......Page 135
Part V: Plan and Design Tests......Page 137
How to Use This Chapter......Page 138
Identify the Objectives......Page 139
Determine Key Usage Scenarios......Page 140
Considerations......Page 141
Considerations......Page 142
Determine Individual User Data and Variances......Page 143
Determine the Relative Distribution of Scenarios......Page 144
Quantifying the Volume of Application Usage: Theory......Page 147
Integrating Model Variance......Page 149
Considerations......Page 150
Summary......Page 151
User Delays......Page 152
Consequences of Improperly Modeling User Delays......Page 153
Step 1 – Determine User Delays......Page 154
Step 2 – Apply Delay Ranges......Page 155
Normal Distribution......Page 156
Negative Exponential Distribution......Page 157
Double Hump Normal Distribution......Page 158
Considerations......Page 159
Considerations......Page 160
Summary......Page 161
Part VI: Execute Tests......Page 162
Validate the Test Environment......Page 163
Additional Considerations......Page 164
Validate Tests......Page 165
Dynamic Data......Page 166
Additional Considerations......Page 167
Run Tests......Page 168
Additional Considerations......Page 169
Baselines......Page 170
Benchmarking......Page 171
Summary......Page 172
Part VII: Analyze Results and Report......Page 173
Exemplar Data Sets......Page 174
Data Set A......Page 175
Data Set C......Page 176
Percentiles......Page 177
Uniform Distributions......Page 178
Normal Distributions......Page 179
Statistical Significance......Page 180
Statistical Outliers......Page 181
Confidence Intervals......Page 182
Summary......Page 184
Principles of Effective Reporting......Page 185
Use the Right Statistics......Page 186
Additional Considerations......Page 187
Frequently Reported Performance Data......Page 188
Exemplar2......Page 189
Exemplar for Stakeholders......Page 190
Volumes, Capacities, and Rates......Page 191
Component Response Times......Page 192
Trends......Page 193
All Roles......Page 194
Types of Results Sharing......Page 195
Technical Reports......Page 196
Creating a Technical Report......Page 197
Exemplar Tables for Single-Instance Measurements......Page 198
Exemplar Summary Statement......Page 199
Exemplar Results Graph......Page 200
Exemplar Workload Model Graphic......Page 201
Exemplar Observations and Recommendations Statement......Page 202
Summary......Page 203
Part VIII: Performance Testing Techniques......Page 204
How to Use This Chapter......Page 205
Approach for Load Testing......Page 206
Step 1 - Identify Performance Acceptance Criteria......Page 207
Step 2 - Identify Key Scenarios......Page 208
Step 5 - Identify Metrics......Page 209
Step 6 - Design Specific Tests......Page 210
Step 7 - Run Tests......Page 211
Summary......Page 212
Examples of Stress-Related Symptoms......Page 213
Approach for Stress Testing......Page 214
Step 1 - Identify Test Objectives......Page 215
Step 3 - Identify the Workload......Page 216
Step 4 - Identify Metrics......Page 217
Test 1 – Place Order Scenario......Page 218
Usage Scenarios for Stress Testing......Page 219
Exploratory Stress Testing......Page 220
Summary......Page 221