Frank Kane's Taming Big Data with Apache Spark and Python

Cover
Copyright
Credits
About the Author
www.PacktPub.com
Customer Feedback
Table of Contents
Preface
Chapter 1: Getting Started with Spark
Getting set up - installing Python, a JDK, and Spark and its dependencies
Installing Enthought Canopy
Installing the Java Development Kit
Installing Spark
Running Spark code
Installing the MovieLens movie rating dataset
Run your first Spark program - the ratings histogram example
Examining the ratings counter script
Running the ratings counter script
Summary
Chapter 2: Spark Basics and Spark Examples
What is Spark?
Spark is scalable
Spark is fast
Spark is hot
Spark is not that hard
Components of Spark
Using Python with Spark
The Resilient Distributed Dataset (RDD)
What is the RDD?
The SparkContext object
Creating RDDs
Transforming RDDs
Map example
RDD actions
Ratings histogram walk-through
Understanding the code
Setting up the SparkContext object
Loading the data
Extract (MAP) the data we care about
Perform an action - count by value
Sort and display the results
Looking at the ratings-counter script in Canopy
Key/value RDDs and the average friends by age example
Key/value concepts - RDDs can hold key/value pairs
Creating a key/value RDD
What Spark can do with key/value data?
Mapping the values of a key/value RDD
The friends by age example
Parsing (mapping) the input data
Counting up the sum of friends and number of entries per age
Compute averages
Collect and display the results
Running the average friends by age example
Examining the script
Running the code
Filtering RDDs and the minimum temperature by location example
What is filter()
The source data for the minimum temperature by location example
Parse (map) the input data
Filter out all but the TMIN entries
Create (station ID, temperature) key/value pairs
Find minimum temperature by station ID
Collect and print results
Running the minimum temperature example and modifying it for maximums
Examining the min-temperatures script
Running the script
Running the maximum temperature by location example
Counting word occurrences using flatmap()
Map versus flatmap
Map ()
Flatmap ()
Code sample - count the words in a book
Improving the word-count script with regular expressions
Text normalization
Examining the use of regular expressions in the word-count script
Running the code
Sorting the word count results
Step 1 - Implement countByValue() the hard way to create a new RDD
Step 2 - Sort the new RDD
Examining the script
Running the code
Find the total amount spent by customer
Introducing the problem
Strategy for solving the problem
Useful snippets of code
Check your results and sort them by the total amount spent
Check your sorted implementation and results against mine
Summary
Chapter 3: Advanced Examples of Spark Programs
Finding the most popular movie
Examining the popular-movies script
Getting results
Using broadcast variables to display movie names instead of ID numbers
Introducing broadcast variables
Examining the popular-movies-nicer.py script
Getting results
Finding the most popular superhero in a social graph
Superhero social networks
Input data format
Strategy
Running the script - discover who the most popular superhero is
Mapping input data to (hero ID, number of co-occurrences) per line
Adding up co-occurrence by hero ID
Flipping the (map) RDD to (number, hero ID)
Using max() and looking up the name of the winner
Getting results
Superhero degrees of separation - introducing the breadth-first search algorithm
Degrees of separation
How the breadth-first search algorithm works?
The initial condition of our social graph
First pass through the graph
Second pass through the graph
Third pass through the graph
Final pass through the graph
Accumulators and implementing BFS in Spark
Convert the input file into structured data
Writing code to convert Marvel-Graph.txt to BFS nodes
Iteratively process the RDD
Using a mapper and a reducer
How do we know when we're done?
Superhero degrees of separation - review the code and run it
Setting up an accumulator and using the convert to BFS function
Calling flatMap()
Calling an action
Calling reduceByKey
Getting results
Item-based collaborative filtering in Spark, cache(), and persist()
How does item-based collaborative filtering work?
Making item-based collaborative filtering a Spark problem
It's getting real
Caching RDDs
Running the similar-movies script using Spark's cluster manager
Examining the script
Getting results
Improving the quality of the similar movies example
Summary
Chapter 4: Running Spark on a Cluster
Introducing Elastic MapReduce
Why use Elastic MapReduce?
Warning - Spark on EMR is not cheap
Setting up our Amazon Web Services / Elastic MapReduce account and PuTTY
Partitioning
Using .partitionBy()
Choosing a partition size
Creating similar movies from one million ratings - part 1
Changes to the script
Creating similar movies from one million ratings - part 2
Our strategy
Specifying memory per executor
Specifying a cluster manager
Running on a cluster
Setting up to run the movie-similarities-1m.py script on a cluster
Preparing the script
Creating a cluster
Connecting to the master node using SSH
Running the code
Creating similar movies from one million ratings – part 3
Assessing the results
Terminating the cluster
Troubleshooting Spark on a cluster
More troubleshooting and managing dependencies
Troubleshooting
Managing dependencies
Summary
Chapter 5: SparkSQL, DataFrames, and DataSets
Introducing SparkSQL
Using SparkSQL in Python
More things you can do with DataFrames
Differences between DataFrames and DataSets
Shell access in SparkSQL
User-defined functions (UDFs)
Executing SQL commands and SQL-style functions on a DataFrame
Using SQL-style functions instead of queries
Using DataFrames instead of RDDs
Summary
Chapter 6: Other Spark Technologies and Libraries
Introducing MLlib
MLlib capabilities
Special MLlib data types
For more information on machine learning
Making movie recommendations
Using MLlib to produce movie recommendations
Examining the movie-recommendations-als.py script
Analyzing the ALS recommendations results
Why did we get bad results?
Using DataFrames with MLlib
Examining the spark-linear-regression.py script
Getting results
Spark Streaming and GraphX
What is Spark Streaming?
GraphX
Summary
Chapter 7: Where to Go From Here? – Learning More About Spark and Data Science
Index