Web Scraping with Python: Gathering Data from the Internet

Web scraping with Python – fetching pages and pulling data out with requests and BeautifulSoup.

What is web scraping?

Web scraping means downloading a web page and parsing its HTML to extract the data you need. Useful when there is no API and no handy CSV download.

The main libraries

Most Python scrapers use requests to fetch pages and BeautifulSoup (from bs4) to parse HTML and XML.

Getting started

Install both libraries:

pip install requests beautifulsoup4

Making HTTP requests

First step: fetch the page content.

import requests

url = 'http://example.com'
response = requests.get(url)

# Check if the request was successful
if response.status_code == 200:
    print('Success!')
else:
    print('An error has occurred.')

Parsing HTML with BeautifulSoup

Pass the response text to BeautifulSoup:

from bs4 import BeautifulSoup

soup = BeautifulSoup(response.text, 'html.parser')
print(soup.prettify())

Extracting data

Pull specific elements from the parsed HTML. To grab every headline on a news site:

# Find all elements with the tag 'h1'
for headline in soup.find_all('h1'):
    print(headline.text.strip())

Navigating the HTML tree

BeautifulSoup lets you find elements by tag, class, and other attributes:

# Find the first element with the tag 'h1'
first_headline = soup.find('h1')
print(first_headline.text.strip())

# Find elements with a specific class
for paragraph in soup.find_all('p', class_='story'):
    print(paragraph.text)

Different page structures

Every site marks up its HTML differently, so your parsing logic will change each time. Right-click > Inspect in your browser to see how the data is structured before you write selectors.

Handling dynamic content

Some sites load content with JavaScript after the initial page load. requests and BeautifulSoup only see the static HTML, so for those you may need Selenium or requests-html to render the page first.

Ethics and best practice

• Respect robots.txt: check the site’s rules before scraping.
• Do not hammer the server: space out requests. Too many too fast is rude and may get you blocked.
• Read the terms of service: some sites forbid scraping outright.

Web scraping in action

A complete example – scrape quotes from a demo site:

import requests
from bs4 import BeautifulSoup

url = 'http://quotes.toscrape.com/'
response = requests.get(url)
soup = BeautifulSoup(response.text, 'html.parser')

quotes = soup.find_all('span', class_='text')

for quote in quotes:
    print(quote.text)

That fetches the page and prints every quote it finds. Scrape responsibly – check the rules, throttle your requests, and prefer an official API when one exists.