Scraping responsibly

We’ll look at fetching and parsing HTML from other sites without being a nuisance. Or breaking terms you agreed to. The intro tutorials showed the mechanics; here we focus on ethics, resilience, and legality at a practical level.

Ask if you need to scrape at all

Many sites offer an API, a data export, or a feed. Use those first. Scraping is fragile: one layout change breaks your parser.

Read robots.txt and terms of use

robots.txt tells well-behaved crawlers what paths to avoid. It is not a full legal contract, but ignoring it is a bad look. The site’s terms of use may explicitly forbid automated access. If they do, stop.

Identify yourself and throttle requests

import time
import requests

SESSION = requests.Session()
SESSION.headers["User-Agent"] = "CodeskillLearner/1.0 (learning project; contact@example.com)"

def fetch(url: str) -> str:
    time.sleep(1)  # be polite
    response = SESSION.get(url, timeout=15)
    response.raise_for_status()
    return response.text

Parse HTML with Beautiful Soup

pip install beautifulsoup4

from bs4 import BeautifulSoup

def extract_headings(html: str) -> list[str]:
    soup = BeautifulSoup(html, "html.parser")
    return [h.get_text(strip=True) for h in soup.select("h2")]

Prefer stable selectors where possible – semantic classes tied to content, not auto-generated CSS module hashes that change every deploy.

Handle failures gracefully

def safe_fetch(url: str, retries: int = 3) -> str | None:
    for attempt in range(1, retries + 1):
        try:
            return fetch(url)
        except requests.RequestException as exc:
            log.warning("fetch failed (%s/%s): %s", attempt, retries, exc)
            time.sleep(2 * attempt)
    return None

Cache responses while developing

Save HTML to disk during development so you are not hammering the live site every time you tweak a parser.

from pathlib import Path

cache = Path("cache/page.html")
if cache.exists():
    html = cache.read_text(encoding="utf-8")
else:
    html = fetch("https://example.com/page")
    cache.write_text(html, encoding="utf-8")

Do not store personal data casually

Scraped pages may contain emails, names, and other personal data. GDPR and similar rules still apply. Collect only what you need, keep it secure, and delete it when you are done learning.

Next: pytest fundamentals – writing tests that protect the code you have built so far.