Transfer Learning for Natural Language Processing

Transfer Learning for Natural Language Processing
contents
preface
acknowledgments
about this book
Who should read this book?
Road map
Software requirements
About the code
liveBook discussion forum
about the author
about the cover illustration
Part 1—Introduction and overview
1 What is transfer learning?
1.1 Overview of representative NLP tasks
1.2 Understanding NLP in the context of AI
1.2.1 Artificial intelligence (AI)
1.2.2 Machine learning
1.2.3 Natural language processing (NLP)
1.3 A brief history of NLP advances
1.3.1 General overview
1.3.2 Recent transfer learning advances
1.4 Transfer learning in computer vision
1.4.1 General overview
1.4.2 Pretrained ImageNet models
1.4.3 Fine-tuning pretrained ImageNet models
1.5 Why is NLP transfer learning an exciting topic to study now?
Summary
2 Getting started with baselines: Data preprocessing
2.1 Preprocessing email spam classification example data
2.1.1 Loading and visualizing the Enron corpus
2.1.2 Loading and visualizing the fraudulent email corpus
2.1.3 Converting the email text into numbers
2.2 Preprocessing movie sentiment classification example data
2.3 Generalized linear models
2.3.1 Logistic regression
2.3.2 Support vector machines (SVMs)
Summary
3 Getting started with baselines: Benchmarking and optimization
3.1 Decision-tree-based models
3.1.1 Random forests (RFs)
3.1.2 Gradient-boosting machines (GBMs)
3.2 Neural network models
3.2.1 Embeddings from Language Models (ELMo)
3.2.2 Bidirectional Encoder Representations from Transformers (BERT)
3.3 Optimizing performance
3.3.1 Manual hyperparameter tuning
3.3.2 Systematic hyperparameter tuning
Summary
Part 2—Shallow transfer learning and deep transfer learning with recurrent neural networks (RNNs)
4 Shallow transfer learning for NLP
4.1 Semisupervised learning with pretrained word embeddings
4.2 Semisupervised learning with higher-level representations
4.3 Multitask learning
4.3.1 Problem setup and a shallow neural single-task baseline
4.3.2 Dual-task experiment
4.4 Domain adaptation
Summary
5 Preprocessing data for recurrent neural network deep transfer learning experiments
5.1 Preprocessing tabular column-type classification data
5.1.1 Obtaining and visualizing tabular data
5.1.2 Preprocessing tabular data
5.1.3 Encoding preprocessed data as numbers
5.2 Preprocessing fact-checking example data
5.2.1 Special problem considerations
5.2.2 Loading and visualizing fact-checking data
Summary
6 Deep transfer learning for NLP with recurrent neural networks
6.1 Semantic Inference for the Modeling of Ontologies (SIMOn)
6.1.1 General neural architecture overview
6.1.2 Modeling tabular data
6.1.3 Application of SIMOn to tabular column-type classification data
6.2 Embeddings from Language Models (ELMo)
6.2.1 ELMo bidirectional language modeling
6.2.2 Application to fake news detection
6.3 Universal Language Model Fine-Tuning (ULMFiT)
6.3.1 Target task language model fine-tuning
6.3.2 Target task classifier fine-tuning
Summary
Part 3—Deep transfer learning with transformers and adaptation strategies
7 Deep transfer learning for NLP with the transformer and GPT
7.1 The transformer
7.1.1 An introduction to the transformers library and attention visualization
7.1.2 Self-attention
7.1.3 Residual connections, encoder-decoder attention, and positional encoding
7.1.4 Application of pretrained encoder-decoder to translation
7.2 The Generative Pretrained Transformer
7.2.1 Architecture overview
7.2.2 Transformers pipelines introduction and application to text generation
7.2.3 Application to chatbots
Summary
8 Deep transfer learning for NLP with BERT and multilingual BERT
8.1 Bidirectional Encoder Representations from Transformers (BERT)
8.1.1 Model architecture
8.1.2 Application to question answering
8.1.3 Application to fill in the blanks and next-sentence prediction tasks
8.2 Cross-lingual learning with multilingual BERT (mBERT)
8.2.1 Brief JW300 dataset overview
8.2.2 Transfer mBERT to monolingual Twi data with the pretrained tokenizer
8.2.3 mBERT and tokenizer trained from scratch on monolingual Twi data
Summary
9 ULMFiT and knowledge distillation adaptation strategies
9.1 Gradual unfreezing and discriminative fine-tuning
9.1.1 Pretrained language model fine-tuning
9.1.2 Target task classifier fine-tuning
9.2 Knowledge distillation
9.2.1 Transfer DistilmBERT to monolingual Twi data with pretrained tokenizer
Summary
10 ALBERT, adapters, and multitask adaptation strategies
10.1 Embedding factorization and cross-layer parameter sharing
10.1.1 Fine-tuning pretrained ALBERT on MDSD book reviews
10.2 Multitask fine-tuning
10.2.1 General Language Understanding Dataset (GLUE)
10.2.2 Fine-tuning on a single GLUE task
10.2.3 Sequential adaptation
10.3 Adapters
Summary
11 Conclusions
11.1 Overview of key concepts
11.2 Other emerging research trends
11.2.1 RoBERTa
11.2.2 GPT-3
11.2.3 XLNet
11.2.4 BigBird
11.2.5 Longformer
11.2.6 Reformer
11.2.7 T5
11.2.8 BART
11.2.9 XLM
11.2.10 TAPAS
11.3 Future of transfer learning in NLP
11.4 Ethical and environmental considerations
11.5 Staying up-to-date
11.5.1 Kaggle and Zindi competitions
11.5.2 arXiv
11.5.3 News and social media (Twitter)
11.6 Final words
Summary
Appendix A—Kaggle primer
A.1 Free GPUs with Kaggle kernels
A.2 Competitions, discussion, and blog
Appendix B—Introduction to fundamental deep learning tools
B.1 Stochastic gradient descent
B.2 TensorFlow
B.3 PyTorch
B.4 Keras, fast.ai, and Transformers by Hugging Face
index
A
B
C
D
E
F
G
H
I
J
K
L
M
N
O
P
Q
R
S
T
U
V
W
X
Z