Natural Language Processing: A Textbook With Python Implementation

Preface
Motivation of This Book
Organization and Structure of This Book
Readers of This Book
How to Use This book?
Acknowledgements
About the Book
Contents
About the Author
Abbreviations
Part I: Concepts and Technology
Chapter 1: Natural Language Processing
1.1 Introduction
1.2 Human Language and Intelligence
1.3 Linguistic Levels of Human Language
1.4 Human Language Ambiguity
1.5 A Brief History of NLP
1.5.1 First Stage: Machine Translation (Before 1960s)
1.5.2 Second Stage: Early AI on NLP from 1960s to 1970s
1.5.3 Third Stage: Grammatical Logic on NLP (1970s–1980s)
1.5.4 Fourth Stage: AI and Machine Learning (1980s–2000s)
1.5.5 Fifth Stage: AI, Big Data, and Deep Networks (2010s–Present)
1.6 NLP and AI
1.7 Main Components of NLP
1.8 Natural Language Understanding (NLU)
1.8.1 Speech Recognition
1.8.2 Syntax Analysis
1.8.3 Semantic Analysis
1.8.4 Pragmatic Analysis
1.9 Potential Applications of NLP
1.9.1 Machine Translation (MT)
1.9.2 Information Extraction (IE)
1.9.3 Information Retrieval (IR)
1.9.4 Sentiment Analysis
1.9.5 Question-Answering (Q&A) Chatbots
References
Chapter 2: N-Gram Language Model
2.1 Introduction
2.2 N-Gram Language Model
2.2.1 Basic NLP Terminology
2.2.2 Language Modeling and Chain Rule
2.3 Markov Chain in N-Gram Model
2.4 Live Example: The Adventures of Sherlock Holmes
2.5 Shannon’s Method in N-Gram Model
2.6 Language Model Evaluation and Smoothing Techniques
2.6.1 Perplexity
2.6.2 Extrinsic Evaluation Scheme
2.6.3 Zero Counts Problems
2.6.4 Smoothing Techniques
2.6.5 Laplace (Add-One) Smoothing
2.6.6 Add-k Smoothing
2.6.7 Backoff and Interpolation Smoothing
2.6.8 Good Turing Smoothing
References
Chapter 3: Part-of-Speech (POS) Tagging
3.1 What Is Part-of-Speech (POS)?
3.1.1 Nine Major POS in English Language
3.2 POS Tagging
3.2.1 What Is POS Tagging in Linguistics?
3.2.2 What Is POS Tagging in NLP?
3.2.3 POS Tags Used in the PENN Treebank Project
3.2.4 Why Do We Care About POS in NLP?
3.3 Major Components in NLU
3.3.1 Computational Linguistics and POS
3.3.2 POS and Semantic Meaning
3.3.3 Morphological and Syntactic Definition of POS
3.4 9 Key POS in English
3.4.1 English Word Classes
3.4.2 What Is a Preposition?
3.4.3 What Is a Conjunction?
3.4.4 What Is a Pronoun?
3.4.5 What Is a Verb?
3.5 Different Types of POS Tagset
3.5.1 What Is Tagset?
3.5.2 Ambiguous in POS Tags
3.5.3 POS Tagging Using Knowledge
3.6 Approaches for POS Tagging
3.6.1 Rule-Based Approach POS Tagging
3.6.2 Example of Rule-Based POS Tagging
3.6.3 Example of Stochastic-Based POS Tagging
3.6.4 Hybrid Approach for POS Tagging Using Brill Taggers
3.6.4.1 What Is Transformation-Based Learning?
3.6.4.2 Hybrid POS Tagging: Brill Tagger
3.6.4.3 Learning Brill Tagger Transformations
3.7 Taggers Evaluations
3.7.1 How Good Is an POS Tagging Algorithm?
References
Chapter 4: Syntax and Parsing
4.1 Introduction and Motivation
4.2 Syntax Analysis
4.2.1 What Is Syntax
4.2.2 Syntactic Rules
4.2.3 Common Syntactic Patterns
4.2.4 Importance of Syntax and Parsing in NLP
4.3 Types of Constituents in Sentences
4.3.1 What Is Constituent?
4.3.2 Kinds of Constituents
4.3.3 Noun-Phrase (NP)
4.3.4 Verb-Phrase (VP)
4.3.5 Complexity on Simple Constituents
4.3.6 Verb Phrase Subcategorization
4.3.7 The Role of Lexicon in Parsing
4.3.8 Recursion in Grammar Rules
4.4 Context-Free Grammar (CFG)
4.4.1 What Is Context-Free Language (CFL)?
4.4.2 What Is Context-Free Grammar (CFG)?
4.4.3 Major Components of CFG
4.4.4 Derivations Using CFG
4.5 CFG Parsing
4.5.1 Morphological Parsing
4.5.2 Phonological Parsing
4.5.3 Syntactic Parsing
4.5.4 Parsing as a Kind of Tree Searching
4.5.5 CFG for Fragment of English
4.5.6 Parse Tree for “Play the Piano” for Prior CFG
4.5.7 Top-Down Parser
4.5.8 Bottom-Up Parser
4.5.9 Control of Parsing
4.5.10 Pros and Cons of Top-Down vs. Bottom-Up Parsing
4.5.10.1 Top-Down Parsing Approach
Pros
Cons
4.5.10.2 Bottom-Up Parsing Approach
Pros
Cons
4.6 Lexical and Probabilistic Parsing
4.6.1 Why Using Probabilities in Parsing?
4.6.2 Semantics with Parsing
4.6.3 What Is PCFG?
4.6.4 A Simple Example of PCFG
4.6.5 Using Probabilities for Language Modeling
4.6.6 Limitations for PCFG
4.6.7 The Fix: Lexicalized Parsing
References
Chapter 5: Meaning Representation
5.1 Introduction
5.2 What Is Meaning?
5.3 Meaning Representations
5.4 Semantic Processing
5.5 Common Meaning Representation
5.5.1 First-Order Predicate Calculus (FOPC)
5.5.2 Semantic Networks
5.5.3 Conceptual Dependency Diagram (CDD)
5.5.4 Frame-Based Representation
5.6 Requirements for Meaning Representation
5.6.1 Verifiability
5.6.2 Ambiguity
5.6.3 Vagueness
5.6.4 Canonical Forms
5.6.4.1 What Is Canonical Form?
5.6.4.2 Canonical Form in Meaning Representation
5.6.4.3 Canonical Forms: Pros and Cons
Advantages
Disadvantages
5.7 Inference
5.7.1 What Is Inference?
5.7.2 Example of Inferencing with FOPC
5.8 Fillmore’s Theory of Universal Cases
5.8.1 What Is Fillmore’s Theory of Universal Cases?
5.8.2 Major Case Roles in Fillmore’s Theory
5.8.3 Complications in Case Roles
5.8.3.1 Selectional Restrictions
5.9 First-Order Predicate Calculus
5.9.1 FOPC Representation Scheme
5.9.2 Major Elements of FOPC
5.9.3 Predicate-Argument Structure of FOPC
5.9.4 Meaning Representation Problems in FOPC
5.9.5 Inferencing Using FOPC
References
Chapter 6: Semantic Analysis
6.1 Introduction
6.1.1 What Is Semantic Analysis?
6.1.2 The Importance of Semantic Analysis in NLP
6.1.3 How Human Is Good in Semantic Analysis?
6.2 Lexical Vs Compositional Semantic Analysis
6.2.1 What Is Lexical Semantic Analysis?
6.2.2 What Is Compositional Semantic Analysis?
6.3 Word Senses and Relations
6.3.1 What Is Word Sense?
6.3.2 Types of Lexical Semantics
6.3.2.1 Homonymy
6.3.2.2 Polysemy
6.3.2.3 Metonymy
6.3.2.4 Zeugma Test
6.3.2.5 Synonyms
6.3.2.6 Antonyms
6.3.2.7 Hyponymy and Hypernymy
6.3.2.8 Hyponyms and Instances
6.4 Word Sense Disambiguation
6.4.1 What Is Word Sense Disambiguation (WSD)?
6.4.2 Difficulties in Word Sense Disambiguation
6.4.3 Method for Word Sense Disambiguation
6.5 WordNet and Online Thesauri
6.5.1 What Is WordNet?
6.5.2 What Is Synsets?
6.5.3 Knowledge Structure of WordNet
6.5.4 What Are Major Lexical Relations Captured in WordNet?
6.5.5 Applications of WordNet and Thesauri?
6.6 Other Online Thesauri: MeSH
6.6.1 What Is MeSH?
6.6.2 Uses of the MeSH Ontology
6.7 Word Similarity and Thesaurus Methods
6.8 Introduction
6.8.1 Path-based Similarity
6.8.2 Problems with Path-based Similarity
6.8.3 Information Content Similarity
6.8.4 The Resnik Method
6.8.5 The Dekang Lin Method
6.8.6 The (Extended) Lesk Algorithm
6.9 Distributed Similarity
6.9.1 Distributional Models of Meaning
6.9.2 Word Vectors
6.9.3 Term-Document Matrix
6.9.4 Point-wise Mutual Information (PMI)
6.9.5 Example of Computing PPMI on a Term-Context Matrix
6.9.6 Weighing PMI Techniques
6.9.7 K-Smoothing in PMI Computation
6.9.8 Context and Word Similarity Measurement
6.9.9 Evaluating Similarity
References
Chapter 7: Pragmatic Analysis and Discourse
7.1 Introduction
7.2 Discourse Phenomena
7.2.1 Coreference Resolution
7.2.2 Why Is it Important?
7.2.3 Coherence and Coreference
7.2.3.1 What Is Coherence?
7.2.3.2 What Is Coreference?
7.2.4 Importance of Coreference Relations
7.2.5 Entity-Based Coherence
7.3 Discourse Segmentation
7.3.1 What Is Discourse Segmentation?
7.3.2 Unsupervised Discourse Segmentation
7.3.3 Hearst’s TextTiling Method
7.3.4 TextTiling Algorithm
7.3.5 Supervised Discourse Segmentation
7.4 Discourse Coherence
7.4.1 What Makes a Text Coherent?
7.4.2 What Is Coherence Relation?
7.4.3 Types of Coherence Relations
7.4.4 Hierarchical Structure of Discourse Coherence
7.4.5 Types of Referring Expressions
7.4.6 Features for Filtering Potential Referents
7.4.7 Preferences in Pronoun Interpretation
7.5 Algorithms for Coreference Resolution
7.5.1 Introduction
7.5.2 Hobbs Algorithm
7.5.2.1 What Is Hobbs Algorithm?
7.5.2.2 Hobbs’ Algorithm
7.5.2.3 Example of Using Hobbs’ Algorithm
7.5.2.4 Performance of Hobbs’ Algorithm
7.5.3 Centering Algorithm
7.5.3.1 What Is Centering Algorithm?
7.5.3.2 Part I: Initial Setting
7.5.3.3 Part II: Constraints
7.5.3.4 Part III: Rules and Algorithm
7.5.3.5 Example of Centering Algorithm
7.5.3.6 Performance of Centering Algorithm
7.5.4 Machine Learning Method
7.5.4.1 What is Machine Learning Method?
7.5.4.2 Performance of Log-Linear Model
7.5.4.3 Other Advanced Machine Learning Models
7.6 Evaluation
References
Chapter 8: Transfer Learning and Transformer Technology
8.1 What Is Transfer Learning?
8.2 Motivation of Transfer Learning
8.2.1 Categories of Transfer Learning
8.3 Solutions of Transfer Learning
8.4 Recurrent Neural Network (RNN)
8.4.1 What Is RNN?
8.4.2 Motivation of RNN
8.4.3 RNN Architecture
8.4.4 Long Short-Term Memory (LSTM) Network
8.4.4.1 What Is LSTM?
8.4.4.2 LSTM Architecture
8.4.5 Gate Recurrent Unit (GRU)
8.4.5.1 What Is GRU?
8.4.5.2 GRU Inner Architecture
8.4.6 Bidirectional Recurrent Neural Networks (BRNNs)
8.4.6.1 What Is BRNN?
8.5 Transformer Technology
8.5.1 What Is Transformer?
8.5.2 Transformer Architecture
8.5.2.1 Encoder
8.5.2.2 Decoder
8.5.3 Deep Into Encoder
8.5.3.1 Positional Encoding
8.5.3.2 Self-Attention Mechanism
8.5.3.3 Multi-Head Attention
8.5.3.4 Layer Normalization of Attention Sublayer
8.5.3.5 Feedforward Layer
8.6 BERT
8.6.1 What Is BERT?
8.6.2 Architecture of BERT
8.6.3 Training of BERT
8.6.3.1 Pre-training BERT
8.6.3.2 Next Sentence Prediction (NSP)
8.6.3.3 Fine-tuning BERT
8.7 Other Related Transformer Technology
8.7.1 Transformer-XL
8.7.1.1 Motivation
8.7.1.2 Transformer-XL technology
8.7.2 ALBERT
References
Chapter 9: Major NLP Applications
9.1 Introduction
9.2 Information Retrieval Systems
9.2.1 Introduction to IR Systems
9.2.2 Vector Space Model in IR
9.2.3 Term Distribution Models in IR
9.2.4 Latent Semantic Indexing in IR
9.2.4.1 Query-Likelihood
9.2.4.2 Document-Likelihood
9.2.5 Discourse Segmentation in IR
9.3 Text Summarization Systems
9.3.1 Introduction to Text Summarization Systems
9.3.1.1 Motivation
9.3.1.2 Task Definition
9.3.1.3 Basic Approach
9.3.1.4 Task Goals
9.3.1.5 Task Sub-processes
9.3.2 Text Summarization Datasets
9.3.3 Types of Summarization Systems
9.3.4 Query-Focused Vs Generic Summarization Systems
9.3.4.1 Query-Focused Summarization Systems
9.3.4.2 Generic Summarization Systems
9.3.5 Single and Multiple Document Summarization
9.3.5.1 Single Document Summarization
9.3.5.2 Multiple Document Summarization
9.3.6 Contemporary Text Summarization Systems
9.3.6.1 Contemporary Extractive Text Summarization (ETS) System
9.3.6.2 Graph-Based Method
9.3.6.3 Feature-Based Method
9.3.6.4 Topic Based Method
9.3.6.5 Grammar-Based Method
9.3.6.6 Contemporary Abstractive Text Summarization (ATS) System
9.3.6.7 Aided Summarization Method
9.3.6.8 Contemporary Combined Text Summarization System
9.4 Question-and-Answering Systems
9.4.1 QA System and AI
9.4.1.1 Rule-based QA Systems
9.4.1.2 Information Retrieval (IR)-based QA Systems
9.4.1.3 Neural Network-Based QA Systems
9.4.2 Overview of Industrial QA Systems
9.4.2.1 AliMe QA System
9.4.2.2 Xiao Ice QA System
9.4.2.3 TransferTransfo Conversational Agents
References
Part II: Natural Language Processing Workshops with Python Implementation in 14 Hours
Chapter 10: Workshop#1 Basics of Natural Language Toolkit (Hour 1–2)
10.1 Introduction
10.2 What Is Natural Language Toolkit (NLTK)?
10.3 A Simple Text Tokenization Example Using NLTK
10.4 How to Install NLTK?
10.5 Why Using Python for NLP?
10.6 NLTK with Basic Text Processing in NLP
10.7 Simple Text Analysis with NLTK
10.8 Text Analysis Using Lexical Dispersion Plot
10.8.1 What Is a Lexical Dispersion Plot?
10.8.2 Lexical Dispersion Plot Over Context Using Sense and Sensibility
10.8.3 Lexical Dispersion Plot Over Time Using Inaugural Address Corpus
10.9 Tokenization in NLP with NLTK
10.9.1 What Is Tokenization in NLP?
10.9.2 Different Between Tokenize() vs Split()
10.9.3 Count Distinct Tokens
10.9.4 Lexical Diversity
10.9.4.1 Token Usage Frequency (Lexical Diversity)
10.9.4.2 Word Usage Frequency
10.10 Basic Statistical Tools in NLTK
10.10.1 Frequency Distribution: FreqDist()
10.10.1.1 FreqDist() as Dictionary Object
10.10.1.2 Access FreqDist of Any Token Type
10.10.1.3 Frequency Distribution Plot from NLTK
10.10.2 Rare Words: Hapax
10.10.3 Collocations
10.10.3.1 What Are Collocations?
10.10.3.2 Collocations in NLTK
References
Chapter 11: Workshop#2 N-grams in NLTK and Tokenization in SpaCy (Hour 3–4)
11.1 Introduction
11.2 What Is N-Gram?
11.3 Applications of N-Grams in NLP
11.4 Generation of N-Grams in NLTK
11.5 Generation of N-Grams Statistics
11.6 spaCy in NLP
11.6.1 What Is spaCy?
11.7 How to Install spaCy?
11.8 Tokenization using spaCy
11.8.1 Step 1: Import spaCy Module
11.8.2 Step 2: Load spaCy Module "en_core_web_sm".
11.8.3 Step 3: Open and Read Text File "Adventures_Holmes.txt" Into file_handler "fholmes"
11.8.4 Step 4: Read Adventures of Sherlock Holmes
11.8.5 Step 5: Replace All Newline Symbols
11.8.6 Step 6: Simple Counting
11.8.7 Step 7: Invoke nlp() Method in spaCy
11.8.8 Step 8: Convert Text Document Into Sentence Object
11.8.9 Step 9: Directly Tokenize Text Document
References
Chapter 12: Workshop#3 POS Tagging Using NLTK (Hour 5–6)
12.1 Introduction
12.2 A Revisit on Tokenization with NLTK
12.3 Stemming Using NLTK
12.3.1 What Is Stemming?
12.3.2 Why Stemming?
12.3.3 How to Perform Stemming?
12.3.4 Porter Stemmer
12.3.5 Snowball Stemmer
12.4 Stop-Words Removal with NLTK
12.4.1 What Are Stop-Words?
12.4.2 NLTK Stop-Words List
12.4.3 Try Some Texts
12.4.4 Create Your Own Stop-Words
12.4.4.1 Step 1: Create Own Stop-Word Library List
12.4.4.2 Step 2: Check Object Type and Will See It Has a Simple List
12.4.4.3 Step 3: Study Stop-Word List
12.4.4.4 Step 4: Add New Stop-Word "sampleSW" Using Append()
12.5 Text Analysis with NLTK
12.6 Integration with WordCloud
12.6.1 What Is WordCloud?
12.7 POS Tagging with NLTK
12.7.1 What Is POS Tagging?
12.7.2 Universal POS Tagset
12.7.3 PENN Treebank Tagset (English and Chinese)
12.7.4 Applications of POS Tagging
12.8 Create Own POS Tagger with NLTK
References
Chapter 13: Workshop#4 Semantic Analysis and Word Vectors Using spaCy (Hour 7–8)
13.1 Introduction
13.2 What Are Word Vectors?
13.3 Understanding Word Vectors
13.3.1 Example: A Simple Word Vector
13.4 A Taste of Word Vectors
13.5 Analogies and Vector Operations
13.6 How to Create Word Vectors?
13.7 spaCy Pre-trained Word Vectors
13.8 Similarity Method in Semantic Analysis
13.9 Advanced Semantic Similarity Methods with spaCy
13.9.1 Understanding Semantic Similarity
13.9.2 Euclidian Distance
13.9.3 Cosine Distance and Cosine Similarity
13.9.4 Categorizing Text with Semantic Similarity
13.9.5 Extracting Key Phrases
13.9.6 Extracting and Comparing Named Entities
References
Chapter 14: Workshop#5 Sentiment Analysis and Text Classification with LSTM Using spaCy (Hour 9–10)
14.1 Introduction
14.2 Text Classification with spaCy and LSTM Technology
14.3 Technical Requirements
14.4 Text Classification in a Nutshell
14.4.1 What Is Text Classification?
14.4.2 Text Classification as AI Applications
14.5 Text Classifier with spaCy NLP Pipeline
14.5.1 TextCategorizer Class
14.5.2 Formatting Training Data for the TextCategorizer
14.5.3 System Training
14.5.4 System Testing
14.5.5 Training TextCategorizer for Multi-Label Classification
14.6 Sentiment Analysis with spaCy
14.6.1 IMDB Large Movie Review Dataset
14.6.2 Explore the Dataset
14.6.3 Training the TextClassfier
14.7 Artificial Neural Network in a Nutshell
14.8 An Overview of TensorFlow and Keras
14.9 Sequential Modeling with LSTM Technology
14.10 Keras Tokenizer in NLP
14.10.1 Embedding Words
14.11 Movie Sentiment Analysis with LTSM Using Keras and spaCy
14.11.1 Step 1: Dataset
14.11.2 Step 2: Data and Vocabulary Preparation
14.11.3 Step 3: Implement the Input Layer
14.11.4 Step 4: Implement the Embedding Layer
14.11.5 Step 5: Implement the LSTM Layer
14.11.6 Step 6: Implement the Output Layer
14.11.7 Step 7: System Compilation
14.11.8 Step 8: Model Fitting and Experiment Evaluation
References
Chapter 15: Workshop#6 Transformers with spaCy and TensorFlow (Hour 11–12)
15.1 Introduction
15.2 Technical Requirements
15.3 Transformers and Transfer Learning in a Nutshell
15.4 Why Transformers?
15.5 An Overview of BERT Technology
15.5.1 What Is BERT?
15.5.2 BERT Architecture
15.5.3 BERT Input Format
15.5.4 How to Train BERT?
15.6 Transformers with TensorFlow
15.6.1 HuggingFace Transformers
15.6.2 Using the BERT Tokenizer
15.6.3 Word Vectors in BERT
15.7 Revisit Text Classification Using BERT
15.7.1 Data Preparation
15.7.1.1 Import Related Modules
15.7.1.2 Read emails.csv Datafile
15.7.1.3 Use dropna() to Remove Record with Missing Contents
15.7.2 Start the BERT Model Construction
15.7.2.1 Import BERT Models and Tokenizer
15.7.2.2 Process Input Data with BertTokenizer
15.7.2.3 Double Check Databank to See Whether Data Has
15.7.2.4 Use BERT Tokenizer
15.7.2.5 Define Keras Model Using the Following Lines
15.7.2.6 Perform Model Fitting and Use 1 Epoch to Save Time
15.7.2.7 Review Model Summary
15.8 Transformer Pipeline Technology
15.8.1 Transformer Pipeline for Sentiment Analysis
15.8.2 Transformer Pipeline for QA System
15.9 Transformer and spaCy
References
Chapter 16: Workshop#7 Building Chatbot with TensorFlow and Transformer Technology (Hour 13–14)
16.1 Introduction
16.2 Technical Requirements
16.3 AI Chatbot in a Nutshell
16.3.1 What Is a Chatbot?
16.3.2 What Is a Wake Word in Chatbot?
16.3.2.1 Tailor-Made Wake Word
16.3.2.2 Why Embedded Word Detection?
16.3.3 NLP Components in a Chatbot
16.4 Building Movie Chatbot by Using TensorFlow and Transformer Technology
16.4.1 The Chatbot Dataset
16.4.2 Movie Dialog Preprocessing
16.4.3 Tokenization of Movie Conversation
16.4.4 Filtering and Padding Process
16.4.5 Creation of TensorFlow Movie Dataset Object (mDS)
16.4.6 Calculate Attention Learning Weights
16.4.7 Multi-Head-Attention (MHAttention)
16.4.8 System Implementation
16.4.8.1 Step 1. Implement Masking
16.4.8.2 Step 2. Implement Positional Encoding
16.4.8.3 Step 3. Implement Encoder Layer
16.4.8.4 Step 4. Implement Encoder
16.4.8.5 Step 5. Implement Decoder Layer
16.4.8.6 Step 6. Implement Decoder
16.4.8.7 Step 7. Implement Transformer
16.4.8.8 Step 8. Model Training
16.4.8.9 Step 9. Implement Model Evaluation Function
16.4.8.10 Step 10. Implement Customized Learning Rate
16.4.8.11 Step 11. Compile Chatbot Model
16.4.8.12 Step 12. System Training (Model Fitting)
16.4.8.13 Step 13. System Evaluation and Live Chatting
16.5 Related Works
References
Index