Work through practical examples to unlock the full potential of web scraping with Python and gain valuable insights from high-quality data
Key Features
Build an initial portfolio of web scraping projects with detailed explanations
Grasp Python programming fundamentals related to web scraping and data extraction
Acquire skills to code web scrapers, store data in desired formats, and employ the data pro
Book Description
Web scraping is a powerful tool for extracting data from the web, but it can be daunting for those without a technical background. Designed for novices, this book will help you grasp the fundamentals of web scraping and Python programming, even if you have no prior experience.
Adopting a practical, hands-on approach, this updated edition of Hands-On Web Scraping with Python uses real-world examples and exercises to explain key concepts. Starting with an introduction to web scraping fundamentals and Python programming, you’ll cover a range of scraping techniques, including requests, lxml, pyquery, Scrapy, and Beautiful Soup. You’ll also get to grips with advanced topics such as secure web handling, web APIs, Selenium for web scraping, PDF extraction, regex, data analysis, EDA reports, visualization, and machine learning.
This book emphasizes the importance of learning by doing. Each chapter integrates examples that demonstrate practical techniques and related skills. By the end of this book, you’ll be equipped with the skills to extract data from websites, a solid understanding of web scraping and Python programming, and the confidence to use these skills in your projects for analysis, visualization, and information discovery.
What you will learn
Master web scraping techniques to extract data from real-world websites
Implement popular web scraping libraries such as requests, lxml, Scrapy, and pyquery
Develop advanced skills in web scraping, APIs, PDF extraction, regex, and machine learning
Analyze and visualize data with Pandas and Plotly
Develop a practical portfolio to demonstrate your web scraping skills
Understand best practices and ethical concerns in web scraping and data extraction
Who this book is for
This book is for beginners who want to learn web scraping and data extraction using Python. No prior programming knowledge is required, but a basic understanding of web-related concepts such as websites, browsers, and HTML is assumed. If you enjoy learning by doing and want to build a portfolio of web scraping projects and delve into data-related studies and application, then this book is tailored for your needs.
Author(s): Anish Chapagain
Edition: 2
Publisher: Packt Publishing Pvt Ltd
Year: 2023
Language: English
Pages: 395
Hands-On Web Scraping with Python
Contributors
About the author
About the reviewers
Preface
Who this book is for
What this book covers
To get the most out of this book
Download the example code files
Conventions used
Get in touch
Share Your Thoughts
Download a free PDF copy of this book
Part 1:Python and Web Scraping
1
Web Scraping Fundamentals
Technical requirements
What is web scraping?
Understanding the latest web technologies
HTTP
HTML
XML
JavaScript
CSS
Data-finding techniques used in web pages
HTML source page
Developer tools
Summary
Further reading
2
Python Programming for Data and Web
Technical requirements
Why Python (for web scraping)?
Accessing the WWW with Python
Setting things up
Creating a virtual environment
Installing libraries
Loading URLs
URL handling and operations
requests – Python library
Implementing HTTP methods
GET
POST
Summary
Further reading
Part 2:Beginning Web Scraping
3
Searching and Processing Web Documents
Technical requirements
Introducing XPath and CSS selectors to process markup documents
The Document Object Model (DOM)
XPath
CSS selectors
Using web browser DevTools to access web content
HTML elements and DOM navigation
XPath and CSS selectors using DevTools
Scraping using lxml – a Python library
lxml by example
Web scraping using lxml
Parsing robots.txt and sitemap.xml
The robots.txt file
Sitemaps
Summary
Further reading
4
Scraping Using PyQuery, a jQuery-Like Library for Python
Technical requirements
PyQuery overview
Introducing jQuery
Exploring PyQuery
Installing PyQuery
Loading a web URL
Element traversing, attributes, and pseudo-classes
Iterating using PyQuery
Web scraping using PyQuery
Example 1 – scraping book details
Example 2 – sitemap to CSV
Example 3 – scraping quotes with author details
Summary
Further reading
5
Scraping the Web with Scrapy and Beautiful Soup
Technical requirements
Web parsing using Python
Introducing Beautiful Soup
Installing Beautiful Soup
Exploring Beautiful Soup
Web scraping using Beautiful Soup
Web scraping using Scrapy
Setting up a project
Creating an item
Implementing the spider
Exporting data
Deploying a web crawler
Summary
Further reading
Part 3:Advanced Scraping Concepts
6
Working with the Secure Web
Technical requirements
Exploring secure web content
Form processing
Cookies and sessions
User authentication
HTML
