Python for Data Science for Dummies

Title Page
Copyright Page
Table of Contents
Introduction
About This Book
Foolish Assumptions
Icons Used in This Book
Beyond the Book
Where to Go from Here
Part I Getting Started with Python for Data Science
Chapter 1 Discovering the Match between Data Science and Python
Defining the Sexiest Job of the 21st Century
Considering the emergence of data science
Outlining the core competencies of a data scientist
Linking data science and big data
Understanding the role of programming
Creating the Data Science Pipeline
Preparing the data
Performing exploratory data analysis
Learning from data
Visualizing
Obtaining insights and data products
Understanding Python’s Role in Data Science
Considering the shifting profile of data scientists
Working with a multipurpose, simple, and efficient language
Learning to Use Python Fast
Loading data
Training a model
Viewing a result
Chapter 2 Introducing Python’s Capabilities and Wonders
Why Python?
Grasping Python’s core philosophy
Discovering present and future development goals
Working with Python
Getting a taste of the language
Understanding the need for indentation
Working at the command line or in the IDE
Performing Rapid Prototyping and Experimentation
Considering Speed of Execution
Visualizing Power
Using the Python Ecosystem for Data Science
Accessing scientific tools using SciPy
Performing fundamental scientific computing using NumPy
Performing data analysis using pandas
Implementing machine learning using Scikit‐learn
Plotting the data using matplotlib
Parsing HTML documents using Beautiful Soup
Chapter 3 Setting Up Python for Data Science
Considering the Off‐the‐Shelf Cross‐Platform Scientific Distributions
Getting Continuum Analytics Anaconda
Getting Enthought Canopy Express
Getting pythonxy
Getting WinPython
Installing Anaconda on Windows
Installing Anaconda on Linux
Installing Anaconda on Mac OS X
Downloading the Datasets and Example Code
Using IPython Notebook
Defining the code repository
Understanding the datasets used in this book
Chapter 4 Reviewing Basic Python
Working with Numbers and Logic
Performing variable assignments
Doing arithmetic
Comparing data using Boolean expressions
Creating and Using Strings
Interacting with Dates
Creating and Using Functions
Creating reusable functions
Calling functions in a variety of ways
Using Conditional and Loop Statements
Making decisions using the if statement
Choosing between multiple options using nested decisions
Performing repetitive tasks using for
Using the while statement
Storing Data Using Sets, Lists, and Tuples
Performing operations on sets
Working with lists
Creating and using Tuples
Defining Useful Iterators
Indexing Data Using Dictionaries
Part II Getting Your Hands Dirty with Data
Chapter 5 Working with Real Data
Uploading, Streaming, and Sampling Data
Uploading small amounts of data into memory
Streaming large amounts of data into memory
Sampling data
Accessing Data in Structured Flat‐File Form
Reading from a text file
Reading CSV delimited format
Reading Excel and other Microsoft Office files
Sending Data in Unstructured File Form
Managing Data from Relational Databases
Interacting with Data from NoSQL Databases
Accessing Data from the Web
Chapter 6 Conditioning Your Data
Juggling between NumPy and pandas
Knowing when to use NumPy
Knowing when to use pandas
Validating Your Data
Figuring out what’s in your data
Removing duplicates
Creating a data map and data plan
Manipulating Categorical Variables
Creating categorical variables
Renaming levels
Combining levels
Dealing with Dates in Your Data
Formatting date and time values
Using the right time transformation
Dealing with Missing Data
Finding the missing data
Encoding missingness
Imputing missing data
Slicing and Dicing: Filtering and Selecting Data
Slicing rows
Slicing columns
Dicing
Concatenating and Transforming
Adding new cases and variables
Removing data
Sorting and shuffling
Aggregating Data at Any Level
Chapter 7 Shaping Data
Working with HTML Pages
Parsing XML and HTML
Using XPath for data extraction
Working with Raw Text
Dealing with Unicode
Stemming and removing stop words
Introducing regular expressions
Using the Bag of Words Model and Beyond
Understanding the bag of words model
Working with n‐grams
Implementing TF‐IDF transformations
Working with Graph Data
Understanding the adjacency matrix
Using NetworkX basics
Chapter 8 Putting What You Know in Action
Contextualizing Problems and Data
Evaluating a data science problem
Researching solutions
Formulating a hypothesis
Preparing your data
Considering the Art of Feature Creation
Defining feature creation
Combining variables
Understanding binning and discretization
Using indicator variables
Transforming distributions
Performing Operations on Arrays
Using vectorization
Performing simple arithmetic on vectors and matrices
Performing matrix vector multiplication
Performing matrix multiplication
Part III Visualizing the Invisible
Chapter 9 Getting a Crash Course in MatPlotLib
Starting with a Graph
Defining the plot
Drawing multiple lines and plots
Saving your work
Setting the Axis, Ticks, Grids
Getting the axes
Formatting the axes
Adding grids
Defining the Line Appearance
Working with line styles
Using colors
Adding markers
Using Labels, Annotations, and Legends
Adding labels
Annotating the chart
Creating a legend
Chapter 10 Visualizing the Data
Choosing the Right Graph
Showing parts of a whole with pie charts
Creating comparisons with bar charts
Showing distributions using histograms
Depicting groups using box plots
Seeing data patterns using scatterplots
Creating Advanced Scatterplots
Depicting groups
Showing correlations
Plotting Time Series
Representing time on axes
Plotting trends over time
Plotting Geographical Data
Visualizing Graphs
Developing undirected graphs
Developing directed graphs
Chapter 11 Understanding the Tools
Using the IPython Console
Interacting with screen text
Changing the window appearance
Getting Python help
Getting IPython help
Using magic functions
Discovering objects
Using IPython Notebook
Working with styles
Restarting the kernel
Restoring a checkpoint
Performing Multimedia and Graphic Integration
Embedding plots and other images
Loading examples from online sites
Obtaining online graphics and multimedia
Part IV Wrangling Data
Chapter 12 Stretching Python’s Capabilities
Playing with Scikit‐learn
Understanding classes in Scikit‐learn
Defining applications for data science
Performing the Hashing Trick
Using hash functions
Demonstrating the hashing trick
Working with deterministic selection
Considering Timing and Performance
Benchmarking with timeit
Working with the memory profiler
Running in Parallel
Performing multicore parallelism
Demonstrating multiprocessing
Chapter 13 Exploring Data Analysis
The EDA Approach
Defining Descriptive Statistics for Numeric Data
Measuring central tendency
Measuring variance and range
Working with percentiles
Defining measures of normality
Counting for Categorical Data
Understanding frequencies
Creating contingency tables
Creating Applied Visualization for EDA
Inspecting boxplots
Performing t‐tests after boxplots
Observing parallel coordinates
Graphing distributions
Plotting scatterplots
Understanding Correlation
Using covariance and correlation
Using nonparametric correlation
Considering chi‐square for tables
Modifying Data Distributions
Using the normal distribution
Creating a Z‐score standardization
Transforming other notable distributions
Chapter 14 Reducing Dimensionality
Understanding SVD
Looking for dimensionality reduction
Using SVD to measure the invisible
Performing Factor and Principal Component Analysis
Considering the psychometric model
Looking for hidden factors
Using components, not factors
Achieving dimensionality reduction
Understanding Some Applications
Recognizing faces with PCA
Extracting Topics with NMF
Recommending movies
Chapter 15 Clustering
Clustering with K‐means
Understanding centroid‐based algorithms
Creating an example with image data
Looking for optimal solutions
Clustering big data
Performing Hierarchical Clustering
Moving Beyond the Round-Shaped Clusters: DBScan
Chapter 16 Detecting Outliers in Data
Considering Detection of Outliers
Finding more things that can go wrong
Understanding anomalies and novel data
Examining a Simple Univariate Method
Leveraging on the Gaussian distribution
Making assumptions and checking out
Developing a Multivariate Approach
Using principal component analysis
Using cluster analysis
Automating outliers detection with SVM
Part V Learning from Data
Chapter 17 Exploring Four Simple and Effective Algorithms
Guessing the Number: Linear Regression
Defining the family of linear models
Using more variables
Understanding limitations and problems
Moving to Logistic Regression
Applying logistic regression
Considering when classes are more
Making Things as Simple as Naïve Bayes
Finding out that Naïve Bayes isn’t so naïve
Predicting text classifications
Learning Lazily with Nearest Neighbors
Predicting after observing neighbors
Choosing your k parameter wisely
Chapter 18 Performing Cross‐Validation, Selection, and Optimization
Pondering the Problem of Fitting a Model
Understanding bias and variance
Defining a strategy for picking models
Dividing between training and test sets
Cross‐Validating
Using cross‐validation on k folds
Sampling stratifications for complex data
Selecting Variables Like a Pro
Selecting by univariate measures
Using a greedy search
Pumping Up Your Hyperparameters
Implementing a grid search
Trying a randomized search
Chapter 19 Increasing Complexity with Linear and Nonlinear Tricks
Using Nonlinear Transformations
Doing variable transformations
Creating interactions between variables
Regularizing Linear Models
Relying on Ridge regression (L2)
Using the Lasso (L1)
Leveraging regularization
Combining L1 & L2: Elasticnet
Fighting with Big Data Chunk by Chunk
Determining when there is too much data
Implementing Stochastic Gradient Descent
Understanding Support Vector Machines
Relying on a computational method
Fixing many new parameters
Classifying with SVC
Going nonlinear is easy
Performing regression with SVR
Creating a stochastic solution with SVM
Chapter 20 Understanding the Power of the Many
Starting with a Plain Decision Tree
Understanding a decision tree
Creating classification and regression trees
Making Machine Learning Accessible
Working with a Random Forest classifier
Working with a Random Forest regressor
Optimizing a Random Forest
Boosting Predictions
Knowing that many weak predictors win
Creating a gradient boosting classifier
Creating a gradient boosting regressor
Using GBM hyper‐parameters
Part VI The Part of Tens
Chapter 21 Ten Essential Data Science Resource Collections
Gaining Insights with Data Science Weekly
Obtaining a Resource List at U Climb Higher
Getting a Good Start with KDnuggets
Accessing the Huge List of Resources on Data Science Central
Obtaining the Facts of Open Source Data Science from Masters
Locating Free Learning Resources with Quora
Receiving Help with Advanced Topics at Conductrics
Learning New Tricks from the Aspirational Data Scientist
Finding Data Intelligence and Analytics Resources at AnalyticBridge
Zeroing In on Developer Resources with Jonathan Bower
Chapter 22 Ten Data Challenges You Should Take
Meeting the Data Science London + Scikit‐learn Challenge
Predicting Survival on the Titanic
Finding a Kaggle Competition that Suits Your Needs
Honing Your Overfit Strategies
Trudging Through the MovieLens Dataset
Getting Rid of Spam Emails
Working with Handwritten Information
Working with Pictures
Analyzing Amazon.com Reviews
Interacting with a Huge Graph
Index
EULA