product iamge

Hands-On Web Scraping with Python: Extract quality data from the web using effective Python techniques

This document was uploaded by one of our users. The uploader already confirmed that they had the permission to publish it. If you are author/publisher or own the copyright of this documents, please report to us by using this DMCA report form.

Download
Search on Amazon

Simply click on the Download Book button.

Yes, Book downloads on Ebookily are 100% Free.

Sometimes the book is free on Amazon As well, so go ahead and hit "Search on Amazon"

Work through practical examples to unlock the full potential of web scraping with Python and gain valuable insights from high-quality data

Key Features

  • Build an initial portfolio of web scraping projects with detailed explanations
  • Grasp Python programming fundamentals related to web scraping and data extraction
  • Acquire skills to code web scrapers, store data in desired formats, and employ the data professionally
  • Purchase of the print or Kindle book includes a free PDF eBook

Book Description

Web scraping is a powerful tool for extracting data from the web, but it can be daunting for those without a technical background. Designed for novices, this book will help you grasp the fundamentals of web scraping and Python programming, even if you have no prior experience.

Adopting a practical, hands-on approach, this updated edition of Hands-On Web Scraping with Python uses real-world examples and exercises to explain key concepts. Starting with an introduction to web scraping fundamentals and Python programming, you’ll cover a range of scraping techniques, including requests, lxml, pyquery, Scrapy, and Beautiful Soup. You’ll also get to grips with advanced topics such as secure web handling, web APIs, Selenium for web scraping, PDF extraction, regex, data analysis, EDA reports, visualization, and machine learning.

This book emphasizes the importance of learning by doing. Each chapter integrates examples that demonstrate practical techniques and related skills. By the end of this book, you’ll be equipped with the skills to extract data from websites, a solid understanding of web scraping and Python programming, and the confidence to use these skills in your projects for analysis, visualization, and information discovery.

What you will learn

  • Master web scraping techniques to extract data from real-world websites
  • Implement popular web scraping libraries such as requests, lxml, Scrapy, and pyquery
  • Develop advanced skills in web scraping, APIs, PDF extraction, regex, and machine learning
  • Analyze and visualize data with Pandas and Plotly
  • Develop a practical portfolio to demonstrate your web scraping skills
  • Understand best practices and ethical concerns in web scraping and data extraction

Who this book is for

This book is for beginners who want to learn web scraping and data extraction using Python. No prior programming knowledge is required, but a basic understanding of web-related concepts such as websites, browsers, and HTML is assumed. If you enjoy learning by doing and want to build a portfolio of web scraping projects and delve into data-related studies and application, then this book is tailored for your needs.

Table of Contents

  1. Web Scraping Fundamentals
  2. Python programming for Data and Web
  3. Searching and Processing Web Documents
  4. Scraping Using PyQuery, a jQuery-Like Library for Python
  5. Scraping the Web with Scrapy and Beautiful Soup
  6. Working with the Secure Web
  7. Data Extraction Using Web APIs
  8. Using Selenium to Scrape the Web
  9. Using Regular Expressions and PDFs
  10. Data Mining, Analysis, and Visualization
  11. Machine Learning and Web Scraping
  12. After Scraping – Next Steps and Data Analysis

Author(s): Anish Chapagain
Edition: 2
Publisher: Packt Publishing
Year: 2023

Language: English
Commentary: Publisher PDF | Published: October 2023
Pages: 324
City: Birmingham
Tags: Python; Python Programming; Data Extraction; Web Scraping; Data Scraping; Crawling; Web Crawling; PyQuery; Scrapy; Beautiful Soup; Selenium; Data Mining; Data Analysis; Data Visualization

Cover
Title page
Copyright and Credits
Contributors
Table of Contents
Preface
Part 1:Python and Web Scraping
Chapter 1: Web Scraping Fundamentals
Technical requirements
What is web scraping?
Understanding the latest web technologies
HTTP
HTML
XML
JavaScript
CSS
Data-finding techniques used in web pages
HTML source page
Developer tools
Summary
Further reading
Chapter 2: Python Programming for Data and Web
Technical requirements
Why Python (for web scraping)?
Accessing the WWW with Python
Setting things up
Creating a virtual environment
Installing libraries
Loading URLs
URL handling and operations
requests – Python library
Implementing HTTP methods
GET
POST
Summary
Further reading
Part 2: Beginning Web Scraping
Chapter 3: Searching and Processing Web Documents
Technical requirements
Introducing XPath and CSS selectors to process markup documents
The Document Object Model (DOM)
XPath
CSS selectors
Using web browser DevTools to access web content
HTML elements and DOM navigation
XPath and CSS selectors using DevTools
Scraping using lxml – a Python library
lxml by example
Web scraping using lxml
Parsing robots.txt and sitemap.xml
The robots.txt file
Sitemaps
Summary
Further reading
Chapter 4: Scraping Using PyQuery, a jQuery-Like Library for Python
Technical requirements
PyQuery overview
Introducing jQuery
Exploring PyQuery
Installing PyQuery
Loading a web URL
Element traversing, attributes, and pseudo-classes
Iterating using PyQuery
Web scraping using PyQuery
Example 1 – scraping book details
Example 2 – sitemap to CSV
Example 3 – scraping quotes with author details
Summary
Further reading
Chapter 5: Scraping the Web with Scrapy and Beautiful Soup
Technical requirements
Web parsing using Python
Introducing Beautiful Soup
Installing Beautiful Soup
Exploring Beautiful Soup
Web scraping using Beautiful Soup
Web scraping using Scrapy
Setting up a project
Creating an item
Implementing the spider
Exporting data
Deploying a web crawler
Summary
Further reading
Part 3: Advanced Scraping Concepts
Chapter 6: Working with the Secure Web
Technical requirements
Exploring secure web content
Form processing
Cookies and sessions
User authentication
HTML
processing using Python
User authentication and cookies
Using proxies
Summary
Further reading
Chapter 7: Data Extraction Using Web APIs
Technical requirements
Introduction to web APIs
Types of API
Benefits of web APIs
Data formats and patterns in APIs
Example 1 – sunrise and sunset
Example 2 – GitHub emojis
Example 3 – Open Library
Web scraping using APIs
Example 1 – holidays from the US calendar
Example 2 – Open Library book details
Example 3 – US cities and time zones
Summary
Further reading
Chapter 8: Using Selenium to Scrape the Web
Technical requirements
Introduction to Selenium
Advantages and disadvantages of Selenium
Use cases of Selenium
Components of Selenium
Using Selenium WebDriver
Setting things up
Exploring Selenium
Scraping using Selenium
Example 1 – book information
Example 2 – forms and searching
Summary
Further reading
Chapter 9: Using Regular Expressions and PDFs
Technical requirements
Overview of regex
Regex with Python
re (search, match, and findall)
re.split
re.sub
re.compile
Regex flags
Using regex to extract data
Example 1 – Yamaha dealer information
Example 2 – data from sitemap
Example 3 – Godfrey’s dealer
Data extraction from a PDF
The PyPDF2 library
Extraction using PyPDF2
Summary
Further reading
Part 4: Advanced Data-Related Concepts
Chapter 10: Data Mining, Analysis, and Visualization
Technical requirements
Introduction to data mining
Predictive data mining
Descriptive data mining
Handling collected data
Basic file handling
JSON
CSV
SQLite
Data analysis and visualization
Exploratory Data Analysis using ydata_profiling
pandas and plotly
Summary
Further reading
Chapter 11: Machine Learning and Web Scraping
Technical requirements
Introduction to ML
ML and Python programming
Types of ML
ML using scikit-learn
Simple linear regression
Multiple linear regression
Sentiment analysis
Summary
Further reading
Part 5: Conclusion
Chapter 12: After Scraping – Next Steps and Data Analysis
Technical requirements
What happens after scraping?
Web requests
pycurl
Proxies
Data processing
PySpark
polars
Jobs and careers
Summary
Further reading
Index
Other Books You May Enjoy