This book provides a comprehensive discussion and new insights about linear optimization of content metrics to improve the automatic Evaluation of Text Summaries (ETS). The reader is first introduced to the background and fundamentals of the ETS. Afterward, state-of-the-art evaluation methods that require or do not require human references are described. Based on how linear optimization has improved other natural language processing tasks, we developed a new methodology based on genetic algorithms that optimize content metrics linearly. Under this optimization, we propose SECO-SEVA as an automatic evaluation metric available for research purposes. Finally, the text finishes with a consideration of directions in which automatic evaluation could be improved in the future. The information provided in this book is self-contained. Therefore, the reader does not require an exhaustive background in this area. Moreover, we consider this book the first one that deals with the ETS in depth.
Author(s): Jonathan Rojas-Simon, Yulia Ledeneva, Rene Arnulfo Garcia-Hernandez
Series: Studies in Computational Intelligence, 1048
Publisher: Springer
Year: 2022
Language: English
Pages: 221
City: Cham
Preface
Contents
About the Authors
Abbreviations
1 Introduction
1.1 Main Challenges
1.2 Book Objectives
1.3 What We Can Learn from This Book
1.4 Book Organization
References
2 Background of the ETS
2.1 The Role of Evaluation for NLP and NLG
2.2 Evaluation of Text Summaries (ETS)
2.3 Types of Evaluation
2.3.1 Extrinsic Evaluation
2.3.2 Intrinsic Evaluation
2.4 Beginnings of the Evaluation
2.5 Workshops for Evaluating ATS Systems
2.5.1 SUMMAC
2.5.2 NTCIR
2.5.3 DUC
2.5.4 TAC
2.5.5 INEX
2.5.6 MultiLing
2.6 Summary of the Chapter
References
3 Fundamentals of the ETS
3.1 Manual Evaluation
3.1.1 Linguistic-Based Metrics
3.1.2 Content-Based Metrics
3.2 Semi-automatic Evaluation
3.3 Automatic Evaluation
3.3.1 General Process of Automatic Evaluation
3.3.2 Sentence-Level Metrics
3.3.3 ROUGE
3.4 Evaluation Levels
3.4.1 Micro (Summary Level)
3.4.2 Macro (ATS System Level)
3.5 Correlation Indexes
3.5.1 Pearson Correlation
3.5.2 Spearman Correlation
3.5.3 Kendall Tau Correlation
3.6 Genetic Algorithm
3.6.1 Basic Genetic Algorithm Steps
3.6.2 Chromosome’s Representation, Fitness Function, and Genetic Operators
3.7 Summary of the Chapter
References
4 State-of-the-art Automatic Evaluation Methods
4.1 Evaluation of Summaries with Human References
4.1.1 Basic Elements
4.1.2 AutoSummENG and MeMoG
4.1.3 LSA
4.1.4 ROUGE-WE
4.2 Evaluation of Summaries without Human References
4.2.1 ROUGE-C
4.2.2 SIMetrix
4.2.3 FRESA
4.2.4 LSA
4.3 Optimized Evaluation Methods
4.3.1 ROSE
4.3.2 Linear Regression for the ETS
4.4 Summary of the Chapter
References
5 Linear Optimization for Solving Other NLP Tasks
5.1 Identification of LASA Drug Names
5.2 Detection of Source Code Re-use
5.3 Summary of the Chapter
References
6 A Novel Methodology Based on Linear Optimization of Metrics for the ETS
6.1 Dataset Analysis
6.2 Selection of Automatic Metrics
6.2.1 ROUGE-C Metrics
6.2.2 LSA Metrics
6.2.3 SIMetrix Metrics
6.3 Evaluation of Summaries
6.4 Linear Optimization of Automatic Metrics
6.4.1 Solution Representation
6.4.2 Initial Population
6.4.3 Proposed Fitness Function
6.4.4 Selection of Chromosomes
6.4.5 Proposed Crossover Operator
6.4.6 Proposed Mutation Operator
6.4.7 Cataclysmic Mutation in the Proposed GA
6.4.8 Termination Criterion
6.5 Correlation Analysis of Linear Optimization of Evaluation Metrics
6.6 Summary of the Chapter
References
7 Experimenting with Linear Optimization of Metrics for Single-Document Summarization Evaluation
7.1 Tuning GA Parameters
7.2 Micro Evaluation Performance
7.3 Macro Evaluation Performance
7.4 Overall Ranking of Metrics
7.5 Differences Between SECO-SEVA and ROUGE-C-SU4
7.6 Summary of the Chapter
References
8 Experimenting with Linear Optimization of Metrics for Multi-document Summarization Evaluation
8.1 Configuration of Metrics
8.2 Analysis of Metrics
8.3 Experimental Results
8.4 Comparison Between SECO-SEVA and JS1
8.5 Comparison Between SECO-SEVA and ROUGE Metrics
8.6 Summary of the Chapter
References
9 Conclusions and Future Considerations for the ETS
9.1 Conclusions
9.2 Future Considerations
References
Appendix A List of Stopwords in English
Appendix B Experimental Results of Individual Evaluation Metrics
Appendix C SECO-SEVA (SElf-COntent Summary EVAluation)
C.1 Dependencies and Submodules of SECO-SEVA
C.2 How to Run SECO-SEVA?
Summary of the Book
