Mastering Python

Cover
Copyright
Credits
About the Author
About the Reviewers
www.PacktPub.com
Table of Contents
Preface
Chapter 1: Getting Started – One Environment per Project
Creating a virtual Python environment using venv
Creating your first venv
venv arguments
Differences between virtualenv and venv
Bootstrapping pip using ensurepip
ensurepip usage
Manual pip install
Installing C/C++ packages
Debian and Ubuntu
Red Hat, CentOS, and Fedora
OS X
Windows
Summary
Chapter 2: Pythonic Syntax, Common Pitfalls, and Style Guide
Code style – or what is Pythonic code?
Formatting strings – printf-style or str.format?
PEP20, the Zen of Python
Beautiful is better than ugly
Explicit is better than implicit
Simple is better than complex
Flat is better than nested
Sparse is better than dense
Readability counts
Practicality beats purity
Errors should never pass silently
In the face of ambiguity, refuse the temptation to guess
One obvious way to do it
Now is better than never
Hard to explain, easy to explain
Namespaces are one honking great idea
Conclusion
Explaining PEP8
Duck typing
Differences between value and identity comparisons
Loops
Maximum line length
Verifying code quality, pep8, pyflakes, and more
flake8
Pylint
Common pitfalls
Scope matters!
Function arguments
Class properties
Modifying variables in the global scope
Overwriting and/or creating extra built-ins
Modifying while iterating
Catching exceptions – differences between Python 2 and 3
Late binding – be careful with closures
Circular imports
Import collisions
Summary
Chapter 3: Containers and Collections – Storing Data the Right Way
Time complexity – the big O notation
Core collections
list – a mutable list of items
dict – unsorted but a fast map of items
set – like a dict without values
tuple – the immutable list
Advanced collections
ChainMap – the list of dictionaries
counter – keeping track of the most occurring elements
deque – the double ended queue
defaultdict – dictionary with a default value
namedtuple – tuples with field names
enum – a group of constants
OrderedDict – a dictionary where the insertion order matters
heapq – the ordered list
bisect – the sorted list
Summary
Chapter 4: Functional Programming – Readability versus Brevity
Functional programming
list comprehensions
dict comprehensions
set comprehensions
lambda functions
The Y combinator
functools
partial – no need to repeat all arguments every time
reduce – combining pairs into a single result
Implementing a factorial function
Processing trees
itertools
accumulate – reduce with intermediate results
chain – combining multiple results
combinations – combinatorics in Python
permutations – combinations where the order matters
compress – selecting items using a list of Booleans
dropwhile/takewhile – selecting items using a function
count – infinite range with decimal steps
groupby – grouping your sorted iterable
islice – slicing any iterable
Summary
Chapter 5: Decorators – Enabling Code Reuse by Decorating
Decorating functions
Why functools.wraps is important
How are decorators useful?
Memoization using decorators
Decorators with (optional) arguments
Creating decorators using classes
Decorating class functions
Skipping the instance – classmethod and staticmethod
Properties – smart descriptor usage
Decorating classes
Singletons – classes with a single instance
Total ordering – sortable classes the easy way
Useful decorators
Single dispatch – polymorphism in Python
Contextmanager, with statements made easy
Validation, type checks, and conversions
Useless warnings – how to ignore them
Summary
Chapter 6: Generators and Coroutines – Infinity, One Step at a Time
What are generators?
Advantages and disadvantages of generators
Pipelines – an effective use of generators
tee – using an output multiple times
Generating from generators
Context managers
Coroutines
A basic example
Priming
Closing and throwing exceptions
Bidirectional pipelines
Using the state
Summary
Chapter 7: Async IO – Multithreading without Threads
Introducing the asyncio library
The async and await statements
Python 3.4
Python 3.5
Choosing between the 3.4 and 3.5 syntax
A simple example of single-threaded parallel processing
Concepts of asyncio
Futures and tasks
Event loops
Processes
Asynchronous servers and clients
Basic echo server
Summary
Chapter 8: Metaclasses – Making Classes (Not Instances) Smarter
Dynamically creating classes
A basic metaclass
Arguments to metaclasses
Accessing metaclass attributes through classes
Abstract classes using collections.abc
Internal workings of the abstract classes
Custom type checks
Using abc.ABC before Python 3.4
Automatically registering a plugin system
Importing plugins on-demand
Importing plugins through configuration
Importing plugins through the file system
Order of operations when instantiating classes
Finding the metaclass
Preparing the namespace
Executing the class body
Creating the class object (not instance)
Executing the class decorators
Creating the class instance
Example
Storing class attributes in definition order
The classic solution without metaclasses
Using metaclasses to get a sorted namespace
Summary
Chapter 9: Documentation – How to Use Sphinx and reStructuredText
The reStructuredText syntax
Getting started with reStructuredText
Inline markup
Headers
Lists
Enumerated list
Bulleted list
Option list
Definition list
Nested lists
Links, references, and labels
Images
Substitutions
Blocks, code, math, comments, and quotes
Conclusion
The Sphinx documentation generator
Getting started with Sphinx
Using sphinx-quickstart
Using sphinx-apidoc
Sphinx directives
The table of contents tree directive (toctree)
Autodoc, documenting Python modules, classes, and functions
Sphinx roles
Documenting code
Documenting a class with the Sphinx style
Documenting a class with the Google style
Documenting a class with the NumPy style
Which style to choose
Summary
Chapter 10: Testing and Logging – Preparing for Bugs
Using examples as tests with doctest
A simple doctest example
Writing doctests
Testing with pure documentation
The doctest flags
True and False versus 1 and 0
Normalizing whitespace
Ellipsis
Doctest quirks
Testing dictionaries
Testing floating-point numbers
Times and durations
Testing with py.test
The difference between the unittest and py.test output
The difference between unittest and py.test tests
Simplifying assertions
Parameterizing tests
Automatic arguments using fixtures
Print statements and logging
Plugins
Mock objects
Using unittest.mock
Using py.test monkeypatch
Logging
Configuration
Basic logging configuration
Dictionary configuration
JSON configuration
Ini file configuration
The network configuration
Logger
Usage
Summary
Chapter 11: Debugging – Solving the Bugs
Non-interactive debugging
Inspecting your script using trace
Debugging using logging
Showing call stack without exceptions
Debugging asyncio
Handling crashes using faulthandler
Interactive debugging
Console on demand
Debugging using pdb
Breakpoints
Catching exceptions
Commands
Debugging using ipdb
Other debuggers
Debugging services
Summary
Chapter 12: Performance – Tracking and Reducing your Memory and CPU Usage
What is performance?
Timeit – comparing code snippet performance
cProfile – finding the slowest components
First profiling run
Calibrating your profiler
Selective profiling using decorators
Using profile statistics
Line profiler
Improving performance
Using the right algorithm
Global interpreter lock
Try versus if
Lists versus generators
String concatenation
Addition versus generators
Map versus generators and list comprehensions
Caching
Lazy imports
Using optimized libraries
Just-in-time compiling
Converting parts of your code to C
Memory usage
Tracemalloc
Memory profiler
Memory leaks
Reducing memory usage
Generators versus lists
Recreating collections versus removing items
Using slots
Performance monitoring
Summary
Chapter 13: Multiprocessing – When a Single CPU Core Is not Enough
Multithreading versus multiprocessing
Hyper-threading versus physical CPU cores
Creating a pool of workers
Sharing data between processes
Remote processes
Distributed processing using multiprocessing
Distributed processing using IPyparallel
ipython_config.py
ipython_kernel_config.py
ipcontroller_config.py
ipengine_config.py
ipcluster_config.py
Summary
Chapter 14: Extensions in C/C++, System Calls, and C/C++ Libraries
Introduction
Do you need C/C++ modules?
Windows
OS X
Linux/Unix
Calling C/C++ with ctypes
Platform-specific libraries
Windows
Linux/Unix
OS X
Making it easy
Calling functions and native types
Complex data structures
Arrays
Gotchas with memory management
CFFI
Complex data structures
Arrays
ABI or API?
CFFI or ctypes?
Native C/C++ extensions
A basic example
C is not Python – size matters
The example explained
static
PyObject*
Parsing arguments
C is not Python – errors are silent or lethal
Calling Python from C – handling complex types
Summary
Chapter 15: Packaging – Creating Your Own Libraries or Applications
Installing packages
Setup parameters
Packages
Entry points
Creating global commands
Custom setup.py commands
Package data
Testing packages
Unittest
py.test
Nosetests
C/C++ extensions
Regular extensions
Cython extensions
Wheels – the new eggs
Distributing to the Python Package Index
Summary
Index