Mastering Concurrency in Python: Create faster programs using concurrency, asynchronous, multithreading, and parallel programming

Title Page
Copyright and Credits
Mastering Concurrency in Python
Dedication
About Packt
Why subscribe?
Packt.com
Contributors
About the author
About the reviewers
Packt is searching for authors like you
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Download the color images
Code in Action
Conventions used
Get in touch
Reviews
Advanced Introduction to Concurrent and Parallel Programming
Technical requirements
What is concurrency?
Concurrent versus sequential
Example 1 – checking whether a non-negative number is prime
Concurrent versus parallel
A quick metaphor
Not everything should be made concurrent
Embarrassingly parallel
Inherently sequential
Example 2 – inherently sequential tasks
I/O bound
The history, present, and future of concurrency
The history of concurrency
The present
The future
A brief overview of mastering concurrency in Python
Why Python?
Setting up your Python environment
General setup
Downloading example code
Summary
Questions
Further reading
Amdahl's Law
Technical requirements
Amdahl's Law
Terminology
Formula and interpretation
The formula for Amdahl's Law
A quick example
Implications
Amdahl's Law's relationship to the law of diminishing returns
How to simulate in Python
Practical applications of Amdahl's Law
Summary
Questions
Further reading
Working with Threads in Python
Technical requirements
The concept of a thread
Threads versus processes
Multithreading
An example in Python
An overview of the threading module
The thread module in Python 2
The threading module in Python 3
Creating a new thread in Python
Starting a thread with the thread module
Starting a thread with the threading module
Synchronizing threads
The concept of thread synchronization
The threading.Lock class
An example in Python
Multithreaded priority queue
A connection between real-life and programmatic queues
The queue module
Queuing in concurrent programming
Multithreaded priority queue
Summary
Questions
Further reading
Using the with Statement in Threads
Technical requirements
Context management
Starting from managing files
The with statement as a context manager
The syntax of the with statement
The with statement in concurrent programming
Example of deadlock handling
Summary
Questions
Further reading
Concurrent Web Requests
Technical requirements
The basics of web requests
HTML
HTTP requests
HTTP status code
The requests module
Making a request in Python
Running a ping test
Concurrent web requests
Spawning multiple threads
Refactoring request logic
The problem of timeout
Support from httpstat.us and simulation in Python
Timeout specifications
Good practices in making web requests
Consider the terms of service and data-collecting policies
Error handling
Update your program regularly
Avoid making a large number of requests
Summary
Questions
Further reading
Working with Processes in Python
Technical requirements
The concept of a process
Processes versus threads
Multiprocessing
Introductory example in Python
An overview of the multiprocessing module
The process class
The Pool class
Determining the current process, waiting, and terminating processes
Determining the current process
Waiting for processes
Terminating processes
Interprocess communication
Message passing for a single worker
Message passing between several workers
Summary
Questions
Further reading
Reduction Operators in Processes
Technical requirements
The concept of reduction operators
Properties of a reduction operator
Examples and non-examples
Example implementation in Python
Real-life applications of concurrent reduction operators
Summary
Questions
Further reading
Concurrent Image Processing
Technical requirements
Image processing fundamentals
Python as an image processing tool
Installing OpenCV and NumPy
Computer image basics
RGB values
Pixels and image files
Coordinates inside an image
OpenCV API
Image processing techniques
Grayscaling
Thresholding
Applying concurrency to image processing
Good concurrent image processing practices
Choosing the correct way (out of many)
Spawning an appropriate number of processes
Processing input/output concurrently
Summary
Questions
Further reading
Introduction to Asynchronous Programming
Technical requirements
A quick analogy
Asynchronous versus other programming models
Asynchronous versus synchronous programming
Asynchronous versus threading and multiprocessing
An example in Python
Summary
Questions
Further reading
Implementing Asynchronous Programming in Python
Technical requirements
The asyncio module
Coroutines, event loops, and futures
Asyncio API
The asyncio framework in action
Asynchronously counting down
A note about blocking functions
Asynchronous prime-checking
Improvements from Python 3.7
Inherently blocking tasks
concurrent.futures as a solution for blocking tasks
Changes in the framework
Examples in Python
Summary
Questions
Further reading
Building Communication Channels with asyncio
Technical requirements
The ecosystem of communication channels
Communication protocol layers
Asynchronous programming for communication channels
Transports and protocols in asyncio
The big picture of asyncio's server client
Python example
Starting a server
Installing Telnet
Simulating a connection channel
Sending messages back to clients
Closing the transports
Client-side communication with aiohttp
Installing aiohttp and aiofiles
Fetching a website's HTML code
Writing files asynchronously
Summary
Questions
Further reading
Deadlocks
Technical requirements
The concept of deadlock
The Dining Philosophers problem
Deadlock in a concurrent system
Python simulation
Approaches to deadlock situations
Implementing ranking among resources
Ignoring locks and sharing resources
An additional note about locks
Concluding note on deadlock solutions
The concept of livelock
Summary
Questions
Further reading
Starvation
Technical requirements
The concept of starvation
What is starvation?
Scheduling
Causes of starvation
Starvation's relationship to deadlock
The readers-writers problem
Problem statement
The first readers-writers problem
The second readers-writers problem
The third readers-writers problem
Solutions to starvation
Summary
Questions
Further reading
Race Conditions
Technical requirements
The concept of race conditions
Critical sections
How race conditions occur
Simulating race conditions in Python
Locks as a solution to race conditions
The effectiveness of locks
Implementation in Python
The downside of locks
Turning a concurrent program sequential
Locks do not lock anything
Race conditions in real life
Security
Operating systems
Networking
Summary
Questions
Further reading
The Global Interpreter Lock
Technical requirements
An introduction to the Global Interpreter Lock
An analysis of memory management in Python
The problem that the GIL addresses
Problems raised by the GIL
The potential removal of the GIL from Python
How to work with the GIL
Implementing multiprocessing, rather than multithreading 
Getting around the GIL with native extensions
Utilizing a different Python interpreter
Summary
Questions
Further reading
Designing Lock-Based and Mutex-Free Concurrent Data Structures
Technical requirements
Lock-based concurrent data structures in Python
LocklessCounter and race conditions
Embedding locks in the data structure of the counter
The concept of scalability
Analysis of the scalability of the counter data structure
Approximate counters as a solution for scalability
The idea behind approximate counters
Implementing approximate counters in Python
A few considerations for approximate counter designs
Mutex-free concurrent data structures in Python
The impossibility of being lock-free in Python
Introduction to the network data structure
Implementing a simple network data structure in Python and race conditions
RCU as a solution
Building on simple data structures
Summary
Questions
Further reading
Memory Models and Operations on Atomic Types
Technical requirements
Python memory model
The components of Python memory manager
Memory model as a labeled directed graph
In the context of concurrency
Atomic operations in Python
What does it mean to be atomic?
The GIL reconsidered
Innate atomicity in Python
Atomic versus nonatomic
Simulation in Python
Summary
Questions
Further reading
Building a Server from Scratch
Technical requirements
Low-level network programming via the socket module
The theory of server-side communication
The API of the socket module
Building a simple echo server
Building a calculator server with the socket module
The underlying calculation logic
Implementing the calculator server
Building a non-blocking server
Analyzing the concurrency of the server
Generators in Python
Asynchronous generators and the send method
Making the server non-blocking
Summary
Questions
Further reading
Testing, Debugging, and Scheduling Concurrent Applications
Technical requirements
Scheduling with APScheduler
Installing APScheduler
Not a scheduling service
APScheduler functionalities
APScheduler API
Scheduler classes
Executor classes
Trigger keywords
Common scheduler methods
Examples in Python
Blocking scheduler
Background scheduler
Executor pool
Running on the cloud
Testing and concurrency in Python
Testing concurrent programs
Unit testing
Static code analysis
Testing programs concurrently
Debugging concurrent programs
Debugging tools and techniques
Debugging and concurrency
Summary
Questions
Further reading
Assessments
Chapter 1
Chapter 2
Chapter 3
Chapter 4
Chapter 5
Chapter 6
Chapter 7
Chapter 8
Chapter 9
Chapter 10
Chapter 11
Chapter 12
Chapter 13
Chapter 14
Chapter 15
Chapter 16
Chapter 17
Chapter 18
Chapter 19
Other Books You May Enjoy
Leave a review - let other readers know what you think