Web Scraping in Python
Learn to parse HTML with XPath and CSS selectors in Python and build scrapy spiders that crawl websites and collect data automatically.
Web scraping in Python is about getting useful information out of web pages and into your own data work. Pages keep that information inside HTML, so the first task is understanding how a document is arranged as a tree of elements, attributes and links. With that picture in place, you write XPath and CSS expressions that point at exactly the text you want, then turn them into automated crawlers with the scrapy library that follow links and gather details page after page.
What You Will Achieve
- Read an HTML document as a tree and locate the elements, attributes and links that hold the data you need
- Write and chain XPath and CSS expressions that extract text and href values from a page
- Use scrapy Selector and Response objects to pull information out of a page’s source
- Build a simple scrapy spider that starts from a set of URLs, follows links and parses each page automatically
Program Overview
HTML Structure and XPath Notation
Sets out why scraping is useful in data science, then shows how HTML documents are built and how XPath moves between their elements.
- The value of web scraping for data science
- HTML tree structure, tags and attributes
- Class names and href links
- Basic XPath notation
Selectors, CSS Locators and Response Objects
Goes deeper into XPath with scrapy selectors, adds CSS Locator syntax, combines the two by chaining, and introduces Response objects for work across several websites.
- XPath attributes, text and chaining
- Scrapy Selector objects
- CSS Locators, wildcards and attribute selection
- Response objects
Crawling the Web with Scrapy Spiders
Turns the selection techniques into scrapy spiders that visit multiple pages, follow links and scrape each page automatically, finishing with a capstone crawler.
- Spider classes and start requests
- Parsing pages and following links
- Capstone crawler
Certificate Awarded (Paid)
Complete the course to earn it.
Job Outlook
Web scraping supports data science work wherever the information an analysis needs sits on websites instead of in a ready-made dataset, and the same HTML parsing skills are useful in web development tasks.
Good for: Python users who want to collect data from websites on their own and prefer learning one scraping library closely through hands-on coding exercises.
Keep in mind: All the scraping is done with scrapy; many of the techniques carry over to BeautifulSoup and Selenium, but the exercises do not use those libraries.
Bottom line: A focused path through HTML structure, XPath and CSS selection that ends with a working scrapy crawler you build yourself.
Frequently Asked Questions
What is XPath used for in web scraping?
XPath is a notation for moving through the elements of an HTML document and pointing at the ones you want to extract. The lessons introduce it early, use it with scrapy selectors, and later combine it with CSS Locators through chaining.
What is a Response object in scrapy?
A Response object behaves much like a Selector, so the same XPath and CSS expressions work on it, but it carries extra tools for taking your scraping across multiple websites.
Can I scrape websites of my own choosing afterwards?
The skills are meant to transfer: once you can read a page's HTML, write selectors and set up a spider that follows links, you can apply the same approach to other sites you want to collect data from. The capstone brings these steps together in one crawler.


There are no reviews yet.