The hand is quicker than the eye. In many cases, so is digital video. Maintaining image quality in bandwidth- and memory-restricted environments is quickly becoming a reality as thriving research delves ever deeper into perceptual coding techniques, which discard superfluous data that humans cannot process or detect. Surveying the topic from a Human Visual System (HVS)-based approach, Digital Video Image Quality and Perceptual Coding outlines the principles, metrics, and standards associated with perceptual coding, as well as the latest techniques and applications. This book is divided broadly into three parts. First, it introduces the fundamental theory, concepts, principles, and techniques underlying the field, such as the basics of compression, HVS modeling, and coding artifacts associated with current well-known techniques. The next section focuses on picture quality assessment criteria; subjective and objective methods and metrics, including vision model based digital video impairment metrics; testing procedures; and international standards regarding image quality. Finally, practical applications come into focus, including digital image and video coder designs based on the HVS as well as post-filtering, restoration, error correction, and concealment techniques. The permeation of digital images and video throughout the world cannot be understated. Nor can the importance of preserving quality while using minimal storage space, and Digital Video Image Quality and Perceptual Coding provides the tools necessary to accomplish this goal. Instructors and lecturers wishing to make use of this work as a textbook can download a presentation of 786 slides in PDF format organized to augment the text. accompany our book (H.R. Wu and K.R. Rao, Digital Video Image Quality and Perceptual Coding, CRC Press (ISBN: 0-8247-2777-0), Nov. 2005) for lecturers or instructor to use for their classes if they use the book.
Author(s): H.R. Wu, K.R. Rao
Series: Signal Processing and Communications
Publisher: CRC Press
Year: 2005
Language: English
Pages: 640
dk3155fm.pdf......Page 1
Digital Video Image Quality and Perceptual Coding......Page 4
Contributors......Page 8
Acknowledgments......Page 10
Preface......Page 12
References......Page 18
Contents......Page 20
Table of Contents......Page 0
Part I: Picture Coding and Human Visual System Fundamentals......Page 35
1.1 Introduction to Digital Picture Coding......Page 36
1.2.1 Digital Image Data......Page 39
1.2.2 Digital Video Data......Page 40
1.2.3 Statistical Analysis......Page 42
1.3.1 Entropy Coding......Page 45
1.3.2 Predictive Coding......Page 46
1.3.3.1 Discrete cosine transform (DCT)......Page 47
1.3.3.2 Discrete wavelet transform (DWT)......Page 51
1.4.2 Optimal Quantizer Design......Page 54
1.4.3 Vector Quantization......Page 57
1.5 Rate-Distortion Theory......Page 58
1.6 Human Visual Systems......Page 59
1.6.1 Contrast Ratio......Page 60
1.6.2 Spatial Frequency......Page 61
1.6.4 Mach Bands......Page 63
1.7.1 JPEG-Still Image Coding Standard......Page 64
1.7.2 MPEG-Video Coding Standards......Page 69
1.8 Summary......Page 73
References......Page 74
2.2 A Brief Overview of the Visual System......Page 77
2.3 Color Vision......Page 79
2.3.1 Colorimetry......Page 80
2.3.2 Color Appearance, Color Order Systems and Color Difference......Page 83
2.4.1 Luminance......Page 87
2.4.2 Perceived Intensity......Page 89
2.5 Spatial Vision and Contrast Sensitivity......Page 91
2.5.1 Acuity and Sampling......Page 92
2.5.2 Contrast Sensitivity......Page 94
2.5.3 Multiple Spatial Frequency Channels......Page 96
2.5.3.2 Pattern detection......Page 97
2.5.3.3 Masking and facilitation......Page 98
2.5.3.4 Nonindependence in spatial frequency and orientation......Page 100
2.5.3.5 Chromatic contrast sensitivity......Page 102
2.5.3.6 Suprathreshold contrast sensitivity......Page 103
2.5.3.7 Image compression and image difference......Page 106
2.6.1 Temporal CSF......Page 107
2.6.2 Apparent Motion......Page 109
2.7.1 Image and Video Quality Research......Page 112
2.8 Conclusions......Page 113
References......Page 114
3.1 Introduction......Page 119
3.2 Blocking Effect......Page 120
3.2.2 Predictive Coded Macroblocks......Page 122
3.3 Basis Image Effect......Page 123
3.3.2 Predictive Coded Macroblocks......Page 124
3.5 Color Bleeding......Page 125
3.6 Staircase Effect......Page 128
3.7 Ringing......Page 129
3.8 Mosaic Patterns......Page 131
3.8.1 Intraframe Coded Macroblocks......Page 132
3.8.2 Predictive-Coded Macroblocks......Page 133
3.9 False Contouring......Page 134
3.10 False Edges......Page 136
3.11 MC Mismatch......Page 138
3.12.1 Ringing-Related Mosquito Effect......Page 140
3.12.2 Mismatch-Related Mosquito Effect......Page 141
3.13 Stationary Area Fluctuations......Page 142
3.14 Chrominance Mismatch......Page 144
3.15 Video Scaling and Field Rate Conversion......Page 145
3.15.1 Video Scaling......Page 146
3.15.2 Field Rate Conversion......Page 147
3.16 Deinterlacing......Page 148
3.16.2 Field Repetition......Page 149
3.16.3.2 Median filters......Page 150
3.17 Summary......Page 151
References......Page 152
Part II: Picture Quality Assessment and Metrics......Page 155
4.1 Introduction......Page 156
4.3 Selection of Test Materials......Page 157
4.4.1 Experts......Page 159
4.5 Experimental Design......Page 160
4.5.1 Test Chamber......Page 161
4.5.2 Common Experimental Mistakes......Page 162
4.6.1 Double Stimulus Impairment Scale Method......Page 163
4.6.2 Double Stimulus Quality Scale Method......Page 168
4.6.3 Comparison Scale Method......Page 172
4.6.4 Single Stimulus Methods......Page 173
4.6.5 Continuous Quality Evaluations......Page 174
4.6.6 Discussion of SSCQE and DSCQS......Page 176
4.6.7 Pitfalls of Different Methods......Page 178
4.7.1 History......Page 181
4.8 Summary......Page 182
References......Page 183
5.1 Introduction......Page 185
5.2 Quality Factors......Page 186
5.3 Metric Classification......Page 187
5.4 Pixel-Based Metrics......Page 189
5.5.1 HVS Modeling Fundamentals......Page 190
5.5.2 Single-Channel Models......Page 193
5.5.3 Multi-Channel Models......Page 194
5.6 The Engineering Approach......Page 195
5.6.1 Full-Reference Metrics......Page 196
5.6.3 No-Reference Metrics......Page 197
NR metrics in the DCT domain......Page 198
Other NR metrics......Page 199
5.7.2 Video Quality Experts Group......Page 200
5.7.3 Limits of Prediction Performance......Page 201
5.8 Conclusions and Perspectives......Page 202
References......Page 203
6.1.2 Introduction......Page 210
6.1.3 Construction of a Picture Quality Scale......Page 211
6.1.3.2 Spatial frequency weighting of errors......Page 212
6.1.3.3 Random errors and disturbances......Page 214
6.1.3.4 Structured and localized errors and disturbances......Page 215
6.1.3.5 Principal component analysis......Page 217
6.1.4 Visual Assessment Tests......Page 218
6.1.4.1 Methods......Page 220
6.1.4.2 Test pictures......Page 221
6.1.5.1 Results of principal component analysis......Page 222
6.1.5.3 Evaluation of PQS......Page 224
6.1.5.4 Generality and robustness of PQS......Page 225
6.1.6.2 Contribution of the distortion factors......Page 226
6.1.7.1 Limitations in applications......Page 227
6.1.7.3 Human vision models and image quality metrics......Page 229
6.1.9 Conclusions......Page 230
6.2 Application of PQS to a Variety of Electronic Images......Page 231
6.2.1.1 Picture spatial resolution and viewing distance......Page 232
6.2.1.2 Constancy of viewing distance......Page 234
6.2.2 Linearization of the Scale......Page 235
6.2.3 Importance of Center Area of Image in Quality Evaluation......Page 237
6.3.2 HDTV and Super HDTV......Page 238
6.3.3 Extra High Quality Images......Page 240
6.3.4 Cellular Phone Type......Page 241
6.4.1 SG9 Recommendations for Quality Assessment......Page 242
6.4.2 J.143......Page 243
6.4.2.3 RR scheme......Page 244
6.4.4 J.133......Page 245
6.4.7 J.148......Page 246
6.5 Conclusion......Page 247
References......Page 248
7.1 Structural Similarity and Image Quality......Page 253
7.2 The Structural SIMilarity (SSIM) Index......Page 256
7.3 Image Quality Assessment Based on the SSIM Index......Page 261
7.4 Discussions......Page 264
References......Page 267
8.1 Introduction......Page 270
8.2 Vision Modeling for Impairment Measurement......Page 274
8.2.1 Color Space Conversion......Page 275
8.2.2 Temporal Filtering......Page 276
8.2.3 Spatial Filtering......Page 277
8.2.4 Contrast Gain Control......Page 278
8.2.5 Detection and Pooling......Page 280
8.2.6.1 Parameterization by vision research experiments......Page 281
8.2.6.2 Parameterization by video quality experiments......Page 282
8.3 Perceptual Blocking Distortion Metric......Page 285
8.3.1 Blocking Dominant Region Segmentation......Page 286
8.3.1.1 Vertical and horizontal block edge detection......Page 288
8.3.1.5 Generation of blocking region map......Page 290
8.3.2 Summation of Distortions in Blocking Dominant Regions......Page 292
8.3.3 Performance Evaluation of the PBDM......Page 293
8.4 Perceptual Ringing Distortion Measure......Page 296
8.4.1 Ringing Region Segmentation......Page 298
8.4.1.2 Smooth and complex region detection......Page 299
8.4.1.3 Boundary labeling and distortion calculation......Page 300
8.4.3 Performance Evaluation of the PRDM......Page 301
8.5 Conclusion......Page 302
References......Page 304
9.1 Introduction......Page 307
9.1.1 Single-Stimulus JND Tests......Page 308
9.1.3 Applications of JND Models......Page 309
9.1.4 Objectives and Organization of the Following Sections......Page 310
9.2 JND with DCT Subbands......Page 311
9.2.1.1 Spatial CSF equations......Page 312
9.2.1.2 Base threshold......Page 313
9.2.2 Luminance Adaptation Considerations......Page 315
9.2.3.2 Inter-band masking......Page 317
9.2.4 Other Factors......Page 319
9.3.1.1 Spatial JNDs......Page 320
9.3.1.2 Simplified estimators......Page 321
9.3.1.3 Temporal masking effect......Page 322
9.3.2.1 Subband summation to pixel domain......Page 323
9.4 JND Model Evaluation......Page 324
9.5 Conclusions......Page 325
References......Page 326
10.1 Introduction......Page 330
10.2 State-of-the-Art for No-Reference Metrics......Page 331
10.3 Quality Metric Components and Design......Page 332
10.3.1 Blocking Artifacts......Page 334
10.3.2 Ringing Artifacts......Page 335
10.3.3 Clipping......Page 336
10.3.5 Contrast......Page 337
10.4 No-Reference Overall Quality Metric......Page 338
10.4.1 Building and Training the NROQM......Page 339
10.5 Performance of the Quality Metric......Page 342
10.5.1 Testing NROQM......Page 343
10.5.2 Test with Expert Viewers......Page 345
10.6 Conclusions and Future Research......Page 346
References......Page 347
11.1 Formation......Page 350
11.3 Phase I......Page 351
11.3.2 The Objective Test Plan......Page 353
11.3.4 Results......Page 354
11.4 Phase II......Page 355
11.4.1 Results......Page 356
11.6 Summary......Page 357
References......Page 358
Part III: Perceptual Coding and Processing of Digital Pictures......Page 359
12.1.1 Outline......Page 360
12.2 Noise Visibility and Visual Masking......Page 361
12.3 Architectures for Perceptual Based Coding......Page 363
12.3.1 Masking Calculations......Page 366
12.3.2.1 Macroblock level control......Page 368
12.3.2.2 Picture level control......Page 369
12.3.3 Look Ahead......Page 371
12.4.1.2 Parameters that affect blockiness visibility......Page 375
12.4.3 Perceptual-Based Scalable Coding Schemes......Page 379
12.5 Salience/Maskability Pre-Processing......Page 380
12.6 Application to Multi-Channel Encoding......Page 381
References......Page 382
13.1 Introduction......Page 384
13.1.1 Watson’s DCTune......Page 385
13.1.3 Höntsch and Karam’s APIC......Page 386
13.1.4 Chou and Li’s Perceptually Tuned Subband Image Coder......Page 388
13.1.6 Zeng et al.’s Point-Wise Extended Visual Masking......Page 389
13.2.1 Coder Structure......Page 391
13.2.2.1 Frequency transform......Page 392
13.2.2.2 CSF......Page 394
13.2.2.3 Masking response......Page 395
13.2.2.5 Overall model......Page 396
13.2.3 EBCOT Adaptation......Page 398
13.3 Model Calibration......Page 400
13.3.2 Generation of Distorted Images......Page 401
13.3.3 Subjective Assessment......Page 402
13.3.4 Arrangements and Apparatus......Page 403
13.3.5 Presentation of Material......Page 404
13.3.6 Grading Scale......Page 405
13.3.7 Results......Page 406
13.3.9 Model Optimization......Page 409
13.3.9.1 Full parametric optimization......Page 412
13.3.9.2 Algorithmic optimization......Page 413
13.3.9.3 Coder optimization......Page 414
13.3.9.4 Remarks......Page 415
13.4 Performance Evaluation......Page 417
13.4.2 Objective Evaluation......Page 418
13.4.3 Objective Results......Page 420
13.4.4.2 Trichotomous FCM......Page 423
13.4.4.3 Assessment arrangements......Page 424
13.4.5.1 PC versus EBCOT-MSE......Page 425
13.4.6 Analysis and Discussion......Page 429
13.5.1 Coding Structure......Page 435
13.5.2 Performance Analysis......Page 437
13.5.2.1 Subjective evaluation......Page 438
13.5.2.3 Discussions......Page 439
13.6 Summary......Page 442
References......Page 443
14.1 Foveated Human Vision and Foveated Image Processing......Page 454
14.2.1 Geometric Methods......Page 457
14.2.2 Filtering Based Methods......Page 459
14.2.3 Multiresolution Methods......Page 461
14.3.1 Foveated Perceptual Weighting Model......Page 463
14.3.2 Embedded Foveation Image Coding......Page 468
14.3.3 Foveation Scalable Video Coding......Page 470
14.4 Discussions......Page 475
References......Page 476
15.1 Introduction......Page 481
15.2 Image Compression and Coding Artifacts......Page 483
15.2.1 Blocking Artifacts......Page 484
15.2.2 Ringing Artifacts......Page 486
15.3 Reduction of Blocking Artifacts......Page 487
15.3.1 Adaptive Postfiltering of Transform Coeffcients......Page 491
15.3.1.1 Consideration of masking effect......Page 493
15.3.1.3 Adaptive filtering......Page 495
15.3.1.4 Quantization constraint......Page 496
15.3.2 Implementation......Page 497
15.3.3 Simulation Results......Page 499
15.3.3.2 Performance evaluation......Page 500
15.4 Reduction of Ringing Artifacts......Page 504
References......Page 506
16.1 Introduction......Page 511
16.2.1 Digital Color Video Formats......Page 512
16.2.2 Color Quantization......Page 513
16.2.3 Analysis of Color Bleeding Distortion......Page 514
16.3 Description of the Post-Processor......Page 517
16.4 Experimental Results — Concluding Remarks......Page 521
References......Page 523
17.1 Introduction to Error Resilient Coding Techniques......Page 525
17.2.1 Temporal Localization......Page 526
17.2.3 Concealment......Page 528
17.2.4 Scalability......Page 531
17.3.2 Temporal Concealment......Page 535
17.3.3 The Boundary Matching Algorithm (BMA)......Page 539
17.3.4 Decoder Motion Vector Estimation (DMVE)......Page 542
17.3.5 Extension of DMVE algorithm......Page 544
17.4 Experimental Procedure......Page 545
17.5 Experimental Results......Page 546
17.6 Conclusions......Page 549
References......Page 563
18.1 Picture Coding Structures......Page 564
18.1.1 Performance Criteria......Page 566
18.1.2 Complete vs. Over-Complete Transforms......Page 570
18.1.3 Decisions Decisions......Page 572
18.2 Vision Modeling Issues......Page 575
18.3 Spatio-Temporal Masking in Video Coding......Page 579
18.4.1 Picture Quality Metrics Design Approaches......Page 580
18.4.2 Alternative Assessment Methods and Issues......Page 581
18.4.3 More Challenges in Picture Quality Assessment......Page 582
18.5.1 Incorporating HVS in Existing Coders......Page 583
18.5.2 HVS Inspired Coders......Page 584
18.5.3 Perceptually Lossless Coding......Page 586
18.7 Summary......Page 587
References......Page 588
Appendix A: VQM Performance Metrics......Page 595
Metric 1: The Pearson linear correlation coefficient between DMOSp and the DMOS......Page 596
Metric 2: Spearman rank order correlation coefficient between DMOSp and the DMOS......Page 600
Metric 4: RMS Error......Page 601
Metric 6: Classification Errors......Page 602
A.4: MATLAB Source Code......Page 603
ANOVA [Mil97,Weia]......Page 611
References......Page 613