Master the art of training vision and large language models with conceptual fundaments and industry-expert guidance. Learn about AWS services and design patterns, with relevant coding examples
Key Features
Learn to develop, train, tune, and apply foundation models with optimized end-to-end pipelines
Explore large-scale distributed training for models and datasets with AWS and SageMaker examples
Evaluate, deploy, and operationalize your custom models with bias detection and pipeline monitoring
Book Description
Foundation models have forever changed machine learning. From BERT to ChatGPT, CLIP to Stable Diffusion, when billions of parameters are combined with large datasets and hundreds to thousands of GPUs, the result is nothing short of record-breaking. The recommendations, advice, and code samples in this book will help you pretrain and fine-tune your own foundation models from scratch on AWS and Amazon SageMaker, while applying them to hundreds of use cases across your organization.
With advice from seasoned AWS and machine learning expert Emily Webber, this book helps you learn everything you need to go from project ideation to dataset preparation, training, evaluation, and deployment for large language, vision, and multimodal models. With step-by-step explanations of essential concepts and practical examples, you'll go from mastering the concept of pretraining to preparing your dataset and model, configuring your environment, training, fine-tuning, evaluating, deploying, and optimizing your foundation models.
You will learn how to apply the scaling laws to distributing your model and dataset over multiple GPUs, remove bias, achieve high throughput, and build deployment pipelines.
By the end of this book, you'll be well equipped to embark on your own project to pretrain and fine-tune the foundation models of the future.
What you will learn
Find the right use cases and datasets for pretraining and fine-tuning
Prepare for large-scale training with custom accelerators and GPUs
Configure environments on AWS and SageMaker to maximize performance
Select hyperparameters based on your model and constraints
Distribute your model and dataset using many types of parallelism
Avoid pitfalls with job restarts, intermittent health checks, and more
Evaluate your model with quantitative and qualitative insights
Deploy your models with runtime improvements and monitoring pipelines
Who this book is for
If you're a machine learning researcher or enthusiast who wants to start a foundation modelling project, this book is for you. Applied scientists, data scientists, machine learning engineers, solution architects, product managers, and students will all benefit from this book. Intermediate Python is a must, along with introductory concepts of cloud computing. A strong understanding of deep learning fundamentals is needed, while advanced topics will be explained. The content covers advanced machine learning and cloud techniques, explaining them in an actionable, easy-to-understand way.
Author(s): Emily Webber; Andrea Olgiati
Publisher: Packt Publishing
Year: 2023
Language: English
Pages: 258
Pretrain Vision and Large Language Models in Python
Foreword
Contributors
About the author
Acknowledgment
About the reviewer
Preface
Who is this book for?
What this book covers
To get the most out of this book
Download the example code files
Conventions used
Get in touch
Share Your Thoughts
Download a free PDF copy of this book
Part 1: Before Pretraining
Chapter 1: An Introduction to Pretraining Foundation Models
The art of pretraining and fine-tuning
The Transformer model architecture and self-attention
State-of-the-art vision and language models
Top vision models as of April 2023
Contrastive pretraining and natural language supervision
Top language models as of April 2023
Language technique spotlight – causal modeling and the scaling laws
Encoders and decoders
Summary
References
Chapter 2: Dataset Preparation: Part One
Finding a dataset and use case for foundation modeling
Top pretraining use cases by industry
Delta – how different is your dataset?
Use the scaling laws to size your datasets
Fundamentals – scaling laws of neural language models
Bias detection and mitigation
Enhancing your dataset – multilingual, multimodal, and augmentations
Summary
References
Chapter 3: Model Preparation
Finding your best base model
Starting with the smallest base model you can
Trade-off – simplicity versus complexity
Finding your pretraining loss function
Pretraining loss functions in vision – ViT and CoCa
Pretraining loss functions in language – Alexa Teacher Model
Changing your pretraining loss function
Solving for your model size
Practical approaches to solving for your model size
Not all scaling laws are created equal
Planning future experiments
Summary
References
Part 2: Configure Your Environment
Chapter 4: Containers and Accelerators on the Cloud
What are accelerators and why do they matter?
Getting ready to use your accelerators
How to use accelerators on AWS – Amazon SageMaker
Optimizing accelerator performance
Hyperparameters
Infrastructure optimizations for accelerators on AWS
Troubleshooting accelerator performance
Summary
References
Chapter 5: Distribution Fundamentals
Understanding key concepts – data and model parallelism
What data parallel is all about
What model parallel is all about
Combining model and data parallel
Distributed training on Amazon SageMaker
Distributed training software
SM DDP
SMP library
Advanced techniques to reduce GPU memory
Tensor parallelism
Optimizer state sharding
Activation checkpointing
Sharded data parallelism
Bringing it all home with examples from models today
Stable Diffusion – data parallelism at scale
GPT-3 – model and data parallelism at scale
Summary
References
Chapter 6: Dataset Preparation: Part Two, the Data Loader
Introducing the data loader in Python
Building and testing your own data loader – a case study from Stable Diffusion
Creating embeddings – tokenizers and other key steps for smart features
Optimizing your data pipeline on Amazon SageMaker
Transforming deep learning datasets at scale on AWS
Summary
References
Part 3: Train Your Model
Chapter 7: Finding the Right Hyperparameters
Hyperparameters – batch size, learning rate, and more
Key hyperparameters in vision and language
Tuning strategies
Hyperparameter tuning for foundation models
Scaling up as a function of world size with SageMaker
Tuning on a sample of your data and updating based on world size
Summary
References
Chapter 8: Large-Scale Training on SageMaker
Optimizing your script for SageMaker training
Importing packages
Argument parsing
Top usability features for SageMaker training
Warm pools for rapid experimentation
SSM and SSH into training instances
Track jobs and experiments to replicate results
Summary
References
Chapter 9: Advanced Training Concepts
Evaluating and improving throughput
Calculating model TFLOPS
Using Flash Attention to speed up your training runs
Speeding up your jobs with compilation
Integrating compilation into your PyTorch scripts
Amazon SageMaker Training Compiler and Neo
Best practices for compilation
Running compiled models on Amazon’s Trainium and Inferentia custom hardware
Solving for an optimal training time
Summary
References
Part 4: Evaluate Your Model
Chapter 10: Fine-Tuning and Evaluating
Fine-tuning for language, text, and everything in between
Fine-tuning a language-only model
Fine-tuning vision-only models
Fine-tuning vision-language models
Evaluating foundation models
Model evaluation metrics for vision
Model evaluation metrics in language
Model evaluation metrics in joint vision-language tasks
Incorporating the human perspective with labeling through SageMaker Ground Truth
Reinforcement learning from human feedback
Summary
References
Chapter 11: Detecting, Mitigating, and Monitoring Bias
Detecting bias in ML models
Detecting bias in large vision and language models
Mitigating bias in vision and language models
Bias mitigation in language – counterfactual data augmentation and fair loss functions
Bias mitigation in vision – reducing correlation dependencies and solving sampling issues
Monitoring bias in ML models
Detecting, mitigating, and monitoring bias with SageMaker Clarify
Summary
References
Chapter 12: How to Deploy Your Model
What is model deployment?
What is the best way to host my model?
Model deployment options on AWS with SageMaker
Why should I shrink my model, and how?
Model compilation
Knowledge distillation
Quantization
Hosting distributed models on SageMaker
Model servers and end-to-end hosting optimizations
Summary
References
Part 5: Deploy Your Model
Chapter 13: Prompt Engineering
Prompt engineering – the art of getting more with less
From few- to zero-shot learning
Text-to-image prompt engineering tips
Image-to-image prompt engineering tips
Upscaling
Masking
Prompting for object-to-image with DreamBooth
Prompting large language models
Instruction fine-tuning
Chain-of-thought prompting
Summarization
Defending against prompt injections and jailbreaking
Advanced techniques – prefix and prompt tuning
Prefix tuning
Prompt tuning
Summary
References
Chapter 14: MLOps for Vision and Language
What is MLOps?
Common MLOps pipelines
Continuous integration and continuous deployment
Model monitoring and human-in-the-loop
MLOps for foundation models
MLOps for vision
AWS offerings for MLOps
A quick introduction to SageMaker Pipelines
Summary
References
Chapter 15: Future Trends in Pretraining Foundation Models
Techniques for building applications for LLMs
Building interactive dialogue apps with open-source stacks
Using RAG to ensure high accuracy in LLM applications
Is generation the new classification?
Human-centered design for building applications with LLMs
Other generative modalities
AWS offerings in foundation models
The future of foundation models
The future of pretraining
Summary
References
Index
Why subscribe?
Other Books You May Enjoy
Packt is searching for authors like you
Share Your Thoughts
