Python: Journey from Novice to Expert

Credits
Preface
What this learning path covers
What you need for this learning path
Who this learning path is for
Reader feedback
Customer support
Downloading the example code
Errata
Piracy
Questions
1. Module 1
1. Introduction and First Steps – Take a Deep Breath
A proper introduction
Enter the Python
About Python
Portability
Coherence
Developer productivity
An extensive library
Software quality
Software integration
Satisfaction and enjoyment
What are the drawbacks?
Who is using Python today?
Setting up the environment
Python 2 versus Python 3 – the great debate
Installing Python
Setting up the Python interpreter
About virtualenv
Your first virtual environment
Your friend, the console
How you can run a Python program
Running Python scripts
Running the Python interactive shell
Running Python as a service
Running Python as a GUI application
How is Python code organized
How do we use modules and packages
Python's execution model
Names and namespaces
Scopes
Object and classes
Guidelines on how to write good code
The Python culture
A note on the IDEs
Summary
2. Built-in Data Types
Everything is an object
Mutable or immutable? That is the question
Numbers
Integers
Booleans
Reals
Complex numbers
Fractions and decimals
Immutable sequences
Strings and bytes
Encoding and decoding strings
Indexing and slicing strings
Tuples
Mutable sequences
Lists
Byte arrays
Set types
Mapping types – dictionaries
The collections module
Named tuples
Defaultdict
ChainMap
Final considerations
Small values caching
How to choose data structures
About indexing and slicing
About the names
Summary
3. Iterating and Making Decisions
Conditional programming
A specialized else: elif
The ternary operator
Looping
The for loop
Iterating over a range
Iterating over a sequence
Iterators and iterables
Iterating over multiple sequences
The while loop
The break and continue statements
A special else clause
Putting this all together
Example 1 – a prime generator
Example 2 – applying discounts
A quick peek at the itertools module
Infinite iterators
Iterators terminating on the shortest input sequence
Combinatoric generators
Summary
4. Functions, the Building Blocks of Code
Why use functions?
Reduce code duplication
Splitting a complex task
Hide implementation details
Improve readability
Improve traceability
Scopes and name resolution
The global and nonlocal statements
Input parameters
Argument passing
Assignment to argument names don't affect the caller
Changing a mutable affects the caller
How to specify input parameters
Positional arguments
Keyword arguments and default values
Variable positional arguments
Variable keyword arguments
Keyword-only arguments
Combining input parameters
Avoid the trap! Mutable defaults
Return values
Returning multiple values
A few useful tips
Recursive functions
Anonymous functions
Function attributes
Built-in functions
One final example
Documenting your code
Importing objects
Relative imports
Summary
5. Saving Time and Memory
map, zip, and filter
map
zip
filter
Comprehensions
Nested comprehensions
Filtering a comprehension
dict comprehensions
set comprehensions
Generators
Generator functions
Going beyond next
The yield from expression
Generator expressions
Some performance considerations
Don't overdo comprehensions and generators
Name localization
Generation behavior in built-ins
One last example
Summary
6. Advanced Concepts – OOP, Decorators, and Iterators
Decorators
A decorator factory
Object-oriented programming
The simplest Python class
Class and object namespaces
Attribute shadowing
I, me, and myself – using the self variable
Initializing an instance
OOP is about code reuse
Inheritance and composition
Accessing a base class
Multiple inheritance
Method resolution order
Static and class methods
Static methods
Class methods
Private methods and name mangling
The property decorator
Operator overloading
Polymorphism – a brief overview
Writing a custom iterator
Summary
7. Testing, Profiling, and Dealing with Exceptions
Testing your application
The anatomy of a test
Testing guidelines
Unit testing
Writing a unit test
Mock objects and patching
Assertions
A classic unit test example
Making a test fail
Interface testing
Comparing tests with and without mocks
Boundaries and granularity
A more interesting example
Test-driven development
Exceptions
Profiling Python
When to profile?
Summary
8. The Edges – GUIs and Scripts
First approach – scripting
The imports
Parsing arguments
The business logic
Second approach – a GUI application
The imports
The layout logic
The business logic
Fetching the web page
Saving the images
Alerting the user
How to improve the application?
Where do we go from here?
The tkinter.tix module
The turtle module
wxPython, PyQt, and PyGTK
The principle of least astonishment
Threading considerations
Summary
9. Data Science
IPython and Jupyter notebook
Dealing with data
Setting up the notebook
Preparing the data
Cleaning the data
Creating the DataFrame
Unpacking the campaign name
Unpacking the user data
Cleaning everything up
Saving the DataFrame to a file
Visualizing the results
Where do we go from here?
Summary
10. Web Development Done Right
What is the Web?
How does the Web work?
The Django web framework
Django design philosophy
The model layer
The view layer
The template layer
The Django URL dispatcher
Regular expressions
A regex website
Setting up Django
Starting the project
Creating users
Adding the Entry model
Customizing the admin panel
Creating the form
Writing the views
The home view
The entry list view
The form view
Tying up URLs and views
Writing the templates
The future of web development
Writing a Flask view
Building a JSON quote server in Falcon
Summary
11. Debugging and Troubleshooting
Debugging techniques
Debugging with print
Debugging with a custom function
Inspecting the traceback
Using the Python debugger
Inspecting log files
Other techniques
Profiling
Assertions
Where to find information
Troubleshooting guidelines
Using console editors
Where to inspect
Using tests to debug
Monitoring
Summary
12. Summing Up – A Complete Example
The challenge
Our implementation
Implementing the Django interface
The setup
The model layer
A simple form
The view layer
Imports and home view
Listing all records
Creating records
Updating records
Deleting records
Setting up the URLs
The template layer
Home and footer templates
Listing all records
Creating and editing records
Talking to the API
Deleting records
Implementing the Falcon API
The main application
Writing the helpers
Coding the password validator
Coding the password generator
Writing the handlers
Coding the password validator handler
Coding the password generator handler
Running the API
Testing the API
Testing the helpers
Testing the handlers
Where do you go from here?
Summary
2. Module 2
1. Object-oriented Design
Introducing object-oriented
Objects and classes
Specifying attributes and behaviors
Data describes objects
Behaviors are actions
Hiding details and creating the public interface
Composition
Inheritance
Inheritance provides abstraction
Multiple inheritance
Case study
Exercises
Summary
2. Objects in Python
Creating Python classes
Adding attributes
Making it do something
Talking to yourself
More arguments
Initializing the object
Explaining yourself
Modules and packages
Organizing the modules
Absolute imports
Relative imports
Organizing module contents
Who can access my data?
Third-party libraries
Case study
Exercises
Summary
3. When Objects Are Alike
Basic inheritance
Extending built-ins
Overriding and super
Multiple inheritance
The diamond problem
Different sets of arguments
Polymorphism
Abstract base classes
Using an abstract base class
Creating an abstract base class
Demystifying the magic
Case study
Exercises
Summary
4. Expecting the Unexpected
Raising exceptions
Raising an exception
The effects of an exception
Handling exceptions
The exception hierarchy
Defining our own exceptions
Case study
Exercises
Summary
5. When to Use Object-oriented Programming
Treat objects as objects
Adding behavior to class data with properties
Properties in detail
Decorators – another way to create properties
Deciding when to use properties
Manager objects
Removing duplicate code
In practice
Case study
Exercises
Summary
6. Python Data Structures
Empty objects
Tuples and named tuples
Named tuples
Dictionaries
Dictionary use cases
Using defaultdict
Counter
Lists
Sorting lists
Sets
Extending built-ins
Queues
FIFO queues
LIFO queues
Priority queues
Case study
Exercises
Summary
7. Python Object-oriented Shortcuts
Python built-in functions
The len() function
Reversed
Enumerate
File I/O
Placing it in context
An alternative to method overloading
Default arguments
Variable argument lists
Unpacking arguments
Functions are objects too
Using functions as attributes
Callable objects
Case study
Exercises
Summary
8. Strings and Serialization
Strings
String manipulation
String formatting
Escaping braces
Keyword arguments
Container lookups
Object lookups
Making it look right
Strings are Unicode
Converting bytes to text
Converting text to bytes
Mutable byte strings
Regular expressions
Matching patterns
Matching a selection of characters
Escaping characters
Matching multiple characters
Grouping patterns together
Getting information from regular expressions
Making repeated regular expressions efficient
Serializing objects
Customizing pickles
Serializing web objects
Case study
Exercises
Summary
9. The Iterator Pattern
Design patterns in brief
Iterators
The iterator protocol
Comprehensions
List comprehensions
Set and dictionary comprehensions
Generator expressions
Generators
Yield items from another iterable
Coroutines
Back to log parsing
Closing coroutines and throwing exceptions
The relationship between coroutines, generators, and functions
Case study
Exercises
Summary
10. Python Design Patterns I
The decorator pattern
A decorator example
Decorators in Python
The observer pattern
An observer example
The strategy pattern
A strategy example
Strategy in Python
The state pattern
A state example
State versus strategy
State transition as coroutines
The singleton pattern
Singleton implementation
The template pattern
A template example
Exercises
Summary
11. Python Design Patterns II
The adapter pattern
The facade pattern
The flyweight pattern
The command pattern
The abstract factory pattern
The composite pattern
Exercises
Summary
12. Testing Object-oriented Programs
Why test?
Test-driven development
Unit testing
Assertion methods
Reducing boilerplate and cleaning up
Organizing and running tests
Ignoring broken tests
Testing with py.test
One way to do setup and cleanup
A completely different way to set up variables
Skipping tests with py.test
Imitating expensive objects
How much testing is enough?
Case study
Implementing it
Exercises
Summary
13. Concurrency
Threads
The many problems with threads
Shared memory
The global interpreter lock
Thread overhead
Multiprocessing
Multiprocessing pools
Queues
The problems with multiprocessing
Futures
AsyncIO
AsyncIO in action
Reading an AsyncIO future
AsyncIO for networking
Using executors to wrap blocking code
Streams
Executors
Case study
Exercises
Summary
3. Module 3
1. Getting Started – One Environment per Project
Creating a virtual Python environment using venv
Creating your first venv
venv arguments
Differences between virtualenv and venv
Bootstrapping pip using ensurepip
ensurepip usage
Manual pip install
Installing C/C++ packages
Debian and Ubuntu
Red Hat, CentOS, and Fedora
OS X
Windows
Summary
2. Pythonic Syntax, Common Pitfalls, and Style Guide
Code style – or what is Pythonic code?
Formatting strings – printf-style or str.format?
PEP20, the Zen of Python
Beautiful is better than ugly
Explicit is better than implicit
Simple is better than complex
Flat is better than nested
Sparse is better than dense
Readability counts
Practicality beats purity
Errors should never pass silently
In the face of ambiguity, refuse the temptation to guess
One obvious way to do it
Now is better than never
Hard to explain, easy to explain
Namespaces are one honking great idea
Conclusion
Explaining PEP8
Duck typing
Differences between value and identity comparisons
Loops
Maximum line length
Verifying code quality, pep8, pyflakes, and more
flake8
Pep8
pyflakes
McCabe
flake8
Pylint
Common pitfalls
Scope matters!
Function arguments
Class properties
Modifying variables in the global scope
Overwriting and/or creating extra built-ins
Modifying while iterating
Catching exceptions – differences between Python 2 and 3
Late binding – be careful with closures
Circular imports
Import collisions
Summary
3. Containers and Collections – Storing Data the Right Way
Time complexity – the big O notation
Core collections
list – a mutable list of items
dict – unsorted but a fast map of items
set – like a dict without values
tuple – the immutable list
Advanced collections
ChainMap – the list of dictionaries
counter – keeping track of the most occurring elements
deque – the double ended queue
defaultdict – dictionary with a default value
namedtuple – tuples with field names
enum – a group of constants
OrderedDict – a dictionary where the insertion order matters
heapq – the ordered list
bisect – the sorted list
Summary
4. Functional Programming – Readability Versus Brevity
Functional programming
list comprehensions
dict comprehensions
set comprehensions
lambda functions
The Y combinator
functools
partial – no need to repeat all arguments every time
reduce – combining pairs into a single result
Implementing a factorial function
Processing trees
itertools
accumulate – reduce with intermediate results
chain – combining multiple results
combinations – combinatorics in Python
permutations – combinations where the order matters
compress – selecting items using a list of Booleans
dropwhile/takewhile – selecting items using a function
count – infinite range with decimal steps
groupby – grouping your sorted iterable
islice – slicing any iterable
Summary
5. Decorators – Enabling Code Reuse by Decorating
Decorating functions
Why functools.wraps is important
How are decorators useful?
Memoization using decorators
Decorators with (optional) arguments
Creating decorators using classes
Decorating class functions
Skipping the instance – classmethod and staticmethod
Properties – smart descriptor usage
Decorating classes
Singletons – classes with a single instance
Total ordering – sortable classes the easy way
Useful decorators
Single dispatch – polymorphism in Python
Contextmanager, with statements made easy
Validation, type checks, and conversions
Useless warnings – how to ignore them
Summary
6. Generators and Coroutines – Infinity, One Step at a Time
What are generators?
Advantages and disadvantages of generators
Pipelines – an effective use of generators
tee – using an output multiple times
Generating from generators
Context managers
Coroutines
A basic example
Priming
Closing and throwing exceptions
Bidirectional pipelines
Using the state
Summary
7. Async IO – Multithreading without Threads
Introducing the asyncio library
The async and await statements
Python 3.4
Python 3.5
Choosing between the 3.4 and 3.5 syntax
A simple example of single-threaded parallel processing
Concepts of asyncio
Futures and tasks
Event loops
Event loop implementations
Event loop policies
Event loop usage
Processes
Asynchronous servers and clients
Basic echo server
Summary
8. Metaclasses – Making Classes (Not Instances) Smarter
Dynamically creating classes
A basic metaclass
Arguments to metaclasses
Accessing metaclass attributes through classes
Abstract classes using collections.abc
Internal workings of the abstract classes
Custom type checks
Using abc.ABC before Python 3.4
Automatically registering a plugin system
Importing plugins on-demand
Importing plugins through configuration
Importing plugins through the file system
Order of operations when instantiating classes
Finding the metaclass
Preparing the namespace
Executing the class body
Creating the class object (not instance)
Executing the class decorators
Creating the class instance
Example
Storing class attributes in definition order
The classic solution without metaclasses
Using metaclasses to get a sorted namespace
Summary
9. Documentation – How to Use Sphinx and reStructuredText
The reStructuredText syntax
Getting started with reStructuredText
Inline markup
Headers
Lists
Enumerated list
Bulleted list
Option list
Definition list
Nested lists
Links, references, and labels
Images
Substitutions
Blocks, code, math, comments, and quotes
Conclusion
The Sphinx documentation generator
Getting started with Sphinx
Using sphinx-quickstart
Using sphinx-apidoc
Sphinx directives
The table of contents tree directive (toctree)
Autodoc, documenting Python modules, classes, and functions
Sphinx roles
Documenting code
Documenting a class with the Sphinx style
Documenting a class with the Google style
Documenting a class with the NumPy style
Which style to choose
Summary
10. Testing and Logging – Preparing for Bugs
Using examples as tests with doctest
A simple doctest example
Writing doctests
Testing with pure documentation
The doctest flags
True and False versus 1 and 0
Normalizing whitespace
Ellipsis
Doctest quirks
Testing dictionaries
Testing floating-point numbers
Times and durations
Testing with py.test
The difference between the unittest and py.test output
The difference between unittest and py.test tests
Simplifying assertions
Parameterizing tests
Automatic arguments using fixtures
Cache
Custom fixtures
Print statements and logging
Plugins
pytest-cov
pytest-pep8 and pytest-flakes
Configuring plugins
Mock objects
Using unittest.mock
Using py.test monkeypatch
Logging
Configuration
Basic logging configuration
Dictionary configuration
JSON configuration
Ini file configuration
The network configuration
Logger
Usage
Summary
11. Debugging – Solving the Bugs
Non-interactive debugging
Inspecting your script using trace
Debugging using logging
Showing call stack without exceptions
Debugging asyncio
Handling crashes using faulthandler
Interactive debugging
Console on demand
Debugging using pdb
Breakpoints
Catching exceptions
Commands
Debugging using ipdb
Other debuggers
Debugging services
Summary
12. Performance – Tracking and Reducing Your Memory and CPU Usage
What is performance?
Timeit – comparing code snippet performance
cProfile – finding the slowest components
First profiling run
Calibrating your profiler
Selective profiling using decorators
Using profile statistics
Line profiler
Improving performance
Using the right algorithm
Global interpreter lock
Try versus if
Lists versus generators
String concatenation
Addition versus generators
Map versus generators and list comprehensions
Caching
Lazy imports
Using optimized libraries
Just-in-time compiling
Converting parts of your code to C
Memory usage
Tracemalloc
Memory profiler
Memory leaks
Reducing memory usage
Generators versus lists
Recreating collections versus removing items
Using slots
Performance monitoring
Summary
13. Multiprocessing – When a Single CPU Core Is Not Enough
Multithreading versus multiprocessing
Hyper-threading versus physical CPU cores
Creating a pool of workers
Sharing data between processes
Remote processes
Distributed processing using multiprocessing
Distributed processing using IPyparallel
ipython_config.py
ipython_kernel_config.py
ipcontroller_config.py
ipengine_config.py
ipcluster_config.py
Summary
14. Extensions in C/C++, System Calls, and C/C++ Libraries
Introduction
Do you need C/C++ modules?
Windows
OS X
Linux/Unix
Calling C/C++ with ctypes
Platform-specific libraries
Windows
Linux/Unix
OS X
Making it easy
Calling functions and native types
Complex data structures
Arrays
Gotchas with memory management
CFFI
Complex data structures
Arrays
ABI or API?
CFFI or ctypes?
Native C/C++ extensions
A basic example
C is not Python – size matters
The example explained
static
PyObject*
Parsing arguments
C is not Python – errors are silent or lethal
Calling Python from C – handling complex types
Summary
15. Packaging – Creating Your Own Libraries or Applications
Installing packages
Setup parameters
Packages
Entry points
Creating global commands
Custom setup.py commands
Package data
Testing packages
Unittest
py.test
Nosetests
C/C++ extensions
Regular extensions
Cython extensions
Wheels – the new eggs
Distributing to the Python Package Index
Summary
A. Bibliography
Index