Measure, optimize, and improve the performance of your Python code with this easy-to-follow guideAbout This Book• Master the do's and don'ts of Python performance programming• Learn how to use exiting new tools that will help you improve your scripts• A step-by-step, conceptual guide to teach you how to optimize and fine-tune your critical pieces of codeWho This Book Is ForIf you're a Python developer looking to improve the speed of your scripts or simply wanting to take your skills to the next level, then this book is perfect for you.What You Will Learn• Master code optimization step-by-step and learn how to use different tools• Understand what a profiler is and how to read its output• Interpret visual output from profiling tools and improve the performance of your script• Use Cython to create fast applications using Python and C• Take advantage of PyPy to improve performance of Python code• Optimize number-crunching code with NumPy, Numba, Parakeet, and PandasIn DetailSimply knowing how to code is not enough; on mission-critical pieces of code, every bit of memory and every CPU cycle counts, and knowing how to squish every bit of processing power out of your code is a crucial and sought-after skill. Nowadays, Python is used for many scientific projects, and sometimes the calculations done in those projects require some serious fine-tuning. Profilers are tools designed to help you measure the performance of your code and help you during the optimization process, so knowing how to use them and read their output is very handy.This book starts from the basics and progressively moves on to more advanced topics. You'll learn everything from profiling all the way up to writing a real-life application and applying a full set of tools designed to improve it in different ways. In the middle, you'll stop to learn about the major profilers used in Python and about some graphic tools to help you make sense of their output. You'll then move from generic optimization techniques onto Python-specific ones, going over the main constructs of the language that will help you improve your speed without much of a change. Finally, the book covers some number-crunching-specific libraries and how to use them properly to get the best speed out of them.After reading this book, you will know how to take any Python code, profile it, find out where the bottlenecks are, and apply different techniques to remove them.Style and approachThis easy-to-follow, practical guide will help you enhance your optimization skills by improving real-world code.
Author(s): Fernando Doglio
Year: 2015
Language: English
Pages: 260
Cover
Copyright
Credits
About the Author
About the Reviewers
www.PacktPub.com
Table of Contents
Preface
Chapter 1: Profiling 101
What is profiling?
Event-based profiling
Statistical profiling
The importance of profiling
What can we profile?
Execution time
Where are the bottlenecks?
Memory consumption and memory leaks
The risk of premature optimization
Running time complexity
Constant time – O(1)
Linear time – O(n)
Logarithmic time – O(log n)
Linearithmic time – O(nlog n)
Factorial time – O(n!)
Quadratic time – O(n^)
Profiling best practices
Build a regression-test suite
Mind your code
Be patient
Gather as much data as you can
Preprocess your data
Visualize your data
Summary
Chapter 2: The Profilers
Getting to know our new best friends: the profilers
cProfile
A note about limitations
The API provided
The Stats class
Profiling examples
Fibonacci again
Tweet stats
line_profiler
kernprof
Some things to consider about kernprof
Profiling examples
Back to Fibonacci
Inverted index
Summary
Chapter 3: Going Visual: GUIs to Help Understand Profiler Output
KCacheGrind – pyprof2calltree
Installation
Usage
A profiling example – TweetStats
A profiling example – Inverted Index
RunSnakeRun
Installation
Usage
Profiling examples – the lowest common multiplier
A profiling example – search using the inverted index
Summary
Chapter 4: Optimize Everything
Memoization / lookup tables
Performing a lookup on a list or linked list
Simple lookup on a dictionary
Binary search
Use cases for lookup tables
Usage of default arguments
List comprehension and generators
ctypes
Loading your own custom C library
Loading a system library
String concatenation
Other tips and tricks
Summary
Chapter 5: Multithreading versus Multiprocessing
Parallelism versus concurrency
Multithreading
Threads
Multiprocessing
Multiprocessing with Python
Summary
Chapter 6: Generic Optimization Options
PyPy
Installing PyPy
A Just-in-time compiler
Sandboxing
Optimizing for the JIT
Think of functions
Consider using cStringIO to concatenate strings
Actions that disable the JIT
Code sample
Cython
Installing Cython
Building a Cython module
Calling C functions
Solving naming conflicts
Defining types
Defining types during function definitions
A Cython example
When to define a type
Limitations
Generator expressions
Comparison of char* literals
Tuples as function arguments
Stack frames
How to choose the right option
When to go with Cython
When to go with PyPy
Summary
Chapter 7: Lightning Fast Number Crunching with Numba, Parakeet, and pandas
Numba
Installation
Using Numba
Numba's code generation
Running your code on the GPU
The pandas tool
Installing pandas
Using pandas for data analysis
Parakeet
Installing Parakeet
How does Parakeet work?
Summary
Chapter 8: Putting It All into Practice
The problem to solve
Getting data from the Web
Postprocessing the data
The initial code base
Analyzing the code
Scraper
Analyzer
Summary
Index
