Text production has many applications. It is used, for instance, to generate dialogue turns from dialogue moves, verbalise the content of knowledge bases, or generate English sentences from rich linguistic representations, such as dependency trees or abstract meaning representations. Text production is also at work in text-to-text transformations such as sentence compression, sentence fusion, paraphrasing, sentence (or text) simplification, and text summarisation. This book offers an overview of the fundamentals of neural models for text production. In particular, we elaborate on three main aspects of neural approaches to text production: how sequential decoders learn to generate adequate text, how encoders learn to produce better input representations, and how neural generators account for task-specific objectives. Indeed, each text-production task raises a slightly different challenge (e.g, how to take the dialogue context into account when producing a dialogue turn, how to detect and merge relevant information when summarising a text, or how to produce a well-formed text that correctly captures the information contained in some input data in the case of data-to-text generation). We outline the constraints specific to some of these tasks and examine how existing neural models account for them. More generally, this book considers text-to-text, meaning-to-text, and data-to-text transformations. It aims to provide the audience with a basic knowledge of neural approaches to text production and a roadmap to get them started with the related work. The book is mainly targeted at researchers, graduate students, and industrials interested in text production from different forms of inputs.
Author(s): Shashi Narayan; Claire Gardent
Series: Synthesis Lectures on Human Language Technologies
Edition: 1
Publisher: Morgan & Claypool
Year: 2020
Language: English
Commentary: Vector PDF
Pages: 199
City: San Rafael, CA
Tags: Deep Learning; Natural Language Processing; Convolutional Neural Networks; Recurrent Neural Networks; Text Generation; Long Short-Term Memory
List of Figures
List of Tables
Preface
Introduction
What is Text Production?
Generating Text from Meaning Representations
Generating Text from Data
Generating Text from Text
Roadmap
What's Not Covered?
Our Notations
Basics
Pre-Neural Approaches
Data-to-Text Generation
Meaning Representations-to-Text Generation
Grammar-Centric Approaches
Statistical MR-to-Text Generation
Text-to-Text Generation
Sentence Simplification and Sentence Compression
Document Summarisation
Summary
Deep Learning Frameworks
Basics
Convolutional Neural Networks
Recurrent Neural Networks
LSTMs and GRUs
Word Embeddings
The Encoder-Decoder Framework
Learning Input Representations with Bidirectional RNNs
Generating Text Using Recurrent Neural Networks
Training and Decoding with Sequential Generators
Differences with Pre-Neural Text-Production Approaches
Summary
Neural Improvements
Generating Better Text
Attention
Copy
Coverage
Summary
Building Better Input Representations
Pitfalls of Modelling Input as a Sequence of Tokens
Modelling Long Text as a Sequence of Tokens
Modelling Graphs or Trees as a Sequence of Tokens
Limitations of Sequential Representation Learning
Modelling Text Structures
Modelling Documents with Hierarchical LSTMs
Modelling Document with Ensemble Encoders
Modelling Document With Convolutional Sentence Encoders
Modelling Graph Structure
Graph-to-Sequence Model for AMR Generation
Graph-Based Triple Encoder for RDF Generation
Graph Convolutional Networks as Graph Encoders
Summary
Modelling Task-Specific Communication Goals
Task-Specific Knowledge for Content Selection
Selective Encoding to Capture Salient Information
Bottom-Up Copy Attention for Content Selection
Graph-Based Attention for Salient Sentence Detection
Multi-Instance and Multi-Task Learning for Content Selection
Optimising Task-Specific Evaluation Metric with Reinforcement Learning
The Pitfalls of Cross-Entropy Loss
Text Production as a Reinforcement Learning Problem
Reinforcement Learning Applications
User Modelling in Neural Conversational Model
Summary
Data Sets and Conclusion
Data Sets and Challenges
Data Sets for Data-to-Text Generation
Generating Biographies from Structured Data
Generating Entity Descriptions from Sets of RDF Triples
Generating Summaries of Sports Games from Box-Score Data
Data Sets for Meaning Representations to Text Generation
Generating from Abstract Meaning Representations
Generating Sentences from Dependency Trees
Generating from Dialogue Moves
Data Sets for Text-to-Text Generation
Summarisation
Simplification
Compression
Paraphrasing
Conclusion
Summarising
Overview of Covered Neural Generators
Two Key Issues with Neural NLG
Challenges
Recent Trends in Neural NLG
Bibliography
Authors' Biographies
Blank Page
