A Complete Guide to Web Crawling, from Core Principles to Real-World Code

To state the conclusion first, crawling is a two-step job: fetch the HTML, then extract only the parts you need. Static pages go with requests, JavaScript pages with Playwright, and large volumes with Scrapy. This covers the principles, the code, and the required programs, step by step.

0. Required Programs: What to Install Before You Start

There are five programs you need for crawling. Install them once and they last.

Python packages go into a virtual environment. Using the system Python directly causes version conflicts.

python3 -m venv crawl-env
source crawl-env/bin/activate
pip install requests beautifulsoup4 lxml httpx scrapy playwright pandas
playwright install chromium

The role of each package is as follows. requests fetches, BeautifulSoup and lxml extract, httpx fetches asynchronously, scrapy is the large-scale framework, playwright is the JS browser, and pandas handles tables (read_html) and CSV processing. playwright install chromium downloads the actual browser, so if you skip this one line the code in section 4 will not run.

To find a selector, right-click your target in the Elements tab of Chrome DevTools (F12) and press Copy selector. If the copied selector is too long, trim it down to the li level.

1. Core Principles: Fetch and Parse

Your code does what the browser does. It sends an HTTP request to the server, receives the HTML, and extracts the desired text from the tag structure.

import requests
from bs4 import BeautifulSoup

headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64)"}
res = requests.get("https://example.com/list", headers=headers, timeout=10)
res.raise_for_status()

soup = BeautifulSoup(res.text, "lxml")
for a in soup.select("ul.news-list > li > a"):
    print(a.get_text(strip=True), a.get("href"))

Three things are key. Without a User-Agent you get blocked as a bot. The CSS selector passed to select is the blueprint for extraction. get_text(strip=True) cleans up the whitespace.

Supplementary Principles: HTTP, HTML Structure, and Encoding

Understanding why crawling works lets you respond when you get stuck. There are four principles.

First, HTTP requests and responses. Typing into the browser address bar is a single GET request. The server returns a status code and HTML. 200 is success, 301 and 302 are moves (redirects), 403 is no entry, 404 is not found, and 429 is too many requests. The crawler reads these numbers and decides its next move.

res = requests.get(url, headers=headers, timeout=10)
print(res.status_code, res.url)

Second, HTML is a tree (the DOM). It is a tree structure where html contains head and body, and body contains ul and li. That is why specifying a branch with a CSS selector lets you pick just the fruit (the text). Selector syntax comes down to five things: tag (li), class (.news), id (#main), child (ul > li), and attribute (a[href]), and these cover 90% of real work.

Third, encoding. If Korean characters come out garbled, it is a res.encoding problem. Older sites without a UTF-8 declaration must be read as euc-kr.

res.encoding = res.apparent_encoding
text = res.text

Fourth, cookies and sessions. Logging in is the server giving the browser a stamp (a cookie). The Session object in requests keeps this stamp, so one login carries the subsequent requests along. The code in section 6 uses exactly this principle.

Fifth, JS rendering. Modern sites serve an empty-shell HTML plus JS, and the browser fills in the content. requests cannot execute JS, so it only receives the empty shell. That is why Playwright works: it is an actual browser.

2. Etiquette and Law: Check robots.txt First

Before crawling, check the target site's allowed crawling scope.

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.set_url("https://example.com/robots.txt")
rp.read()
print(rp.can_fetch("*", "https://example.com/list"))

The principle is simple. Do not touch Disallow paths. Keep the request interval at one second or more. Do not collect information behind a login or personal data. Even for public data, redistribution is a copyright issue, so limit storage to personal research use.

3. Paging Through: Pagination and Infinite Scroll

Half of list crawling is handling the next page. There are two approaches: a URL pattern, and a button click.

import time

base = "https://example.com/list?page={}"
results = []
for page in range(1, 11):
    res = requests.get(base.format(page), headers=headers, timeout=10)
    soup = BeautifulSoup(res.text, "lxml")
    items = soup.select("ul.news-list > li")
    if not items:
        break
    for li in items:
        results.append(li.get_text(strip=True))
    time.sleep(1.2)
print(len(results), "items collected")

The key is to stop when an empty list comes back. Running it without sleep gets your IP blocked.

4. JavaScript Pages: Playwright

Pages built with React and Vue have empty HTML, and JS fills in the content. With requests you only get the empty shell. Playwright, which launches a real browser, solves this.

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(user_agent="Mozilla/5.0 (Windows NT 10.0; Win64; x64)")
    page.goto("https://example.com/dynamic", wait_until="networkidle")
    page.wait_for_selector("ul.news-list > li")
    for li in page.query_selector_all("ul.news-list > li"):
        print(li.inner_text().strip())
    browser.close()

wait_until and wait_for_selector are key. Reading before the content appears gives you an empty result. For infinite scroll, press End on the keyboard to scroll down.

for _ in range(5):
    page.keyboard.press("End")
    page.wait_for_timeout(1500)

5. Large-Scale Collection: Scrapy

For thousands of pages or more, the Scrapy framework is the answer. It provides concurrent requests, retries, and pipelines out of the box.

import scrapy

class NewsSpider(scrapy.Spider):
    name = "news"
    start_urls = ["https://example.com/list?page=1"]
    custom_settings = {"DOWNLOAD_DELAY": 1.0, "CONCURRENT_REQUESTS": 4}

    def parse(self, response):
        for li in response.css("ul.news-list > li"):
            yield {"title": li.css("a::text").get(default="").strip()}
        nxt = response.css("a.next::attr(href)").get()
        if nxt:
            yield response.follow(nxt, callback=self.parse)

Run it with scrapy crawl news -o news.json. DOWNLOAD_DELAY is etiquette, and CONCURRENT_REQUESTS is speed.

6. Speed and Block Evasion: Async and Sessions

Hundreds of pages are handled in parallel asynchronously. The httpx and asyncio combination is the standard.

import asyncio, httpx
from bs4 import BeautifulSoup

async def fetch(client, url):
    r = await client.get(url, timeout=10)
    soup = BeautifulSoup(r.text, "lxml")
    return soup.title.get_text(strip=True)

async def main(urls):
    limits = httpx.Limits(max_connections=5)
    async with httpx.AsyncClient(headers=headers, limits=limits) as client:
        tasks = [fetch(client, u) for u in urls]
        return await asyncio.gather(*tasks, return_exceptions=True)

titles = asyncio.run(main(["https://example.com/1", "https://example.com/2"]))

Five concurrent connections is the safe line. If a login is needed, keep the cookies with a Session.

s = requests.Session()
s.post("https://example.com/login", data={"id": "me", "pw": "secret"})
res = s.get("https://example.com/mypage")

When blocked, tell 429 and 403 apart. A 429 is a signal to take a break and retry after waiting, while a 403 is a structural block, so you have to change your headers and approach.

7. Storage: JSON, CSV, and SQLite

import json, csv, sqlite3

with open("news.json", "w", encoding="utf-8") as f:
    json.dump(results, f, ensure_ascii=False, indent=2)

with open("news.csv", "w", encoding="utf-8-sig", newline="") as f:
    w = csv.writer(f)
    w.writerow(["title"])
    w.writerows([[r] for r in results])

con = sqlite3.connect("news.db")
con.execute("CREATE TABLE IF NOT EXISTS news (title TEXT UNIQUE)")
con.executemany("INSERT OR IGNORE INTO news VALUES (?)", [(r,) for r in results])
con.commit()

CSV uses utf-8-sig for Excel compatibility. UNIQUE plus INSERT OR IGNORE is the simplest way to prevent duplicate collection.

8. Method Selection Table

Eighty percent of crawling skill is selector design and etiquette. One-second intervals against a single site, and backing off when blocked — just these two things keep long-term collection working.