ZerdaZerda

Cookbook: Web Scraping & Crawling Pipeline

On this page

Recipes for turning a list of URLs into indexed documents using WebLoader/MultiURLLoader, respecting rate limits and handling failures gracefully.

Recipe: Polite bulk fetching of a known URL list

Python · 18 lines
from zerda.document_loaders import MultiURLLoader, WebLoaderConfig

urls = [
    "https://example.com/docs/page1",
    "https://example.com/docs/page2",
    "https://example.com/docs/page3",
]

config = WebLoaderConfig(timeout=15, max_retries=3, retry_delay=1.0)
docs = MultiURLLoader(
    urls,
    config=config,
    max_workers=4,
    delay_between_requests=1.0,   # minimum 1s between request starts, globally
    continue_on_error=True,
).load()

print(f"Loaded {len(docs)} documents from {len(urls)} URLs")

Recipe: Single-page fetch with custom headers (e.g. API token)

Python · 7 lines
from zerda.document_loaders import WebLoader

docs = WebLoader(
    "https://internal-wiki.example.com/page/42",
    headers={"Authorization": "Bearer YOUR_TOKEN"},
    timeout=20,
).load()

Recipe: Fetching HTML you already have (from a scraper, browser automation, or an API response)

WebLoader always performs its own HTTP fetch. If you already have HTML text (e.g. from Selenium, Playwright, or a webhook payload), skip straight to HTMLStringLoader instead:

Python · 5 lines
from zerda.document_loaders import HTMLStringLoader, HTMLLoaderConfig

# html_text obtained from your own scraper/automation tool
config = HTMLLoaderConfig(extract_links=True, tags_to_extract=["article", "main"])
docs = HTMLStringLoader(html_text, source="https://example.com/page", config=config).load()

Recipe: Custom HTML extraction for web-fetched pages

WebLoader hardcodes its internal HTMLLoaderConfig (fixed tags_to_remove, extract_links=False, extract_tables=True) — see the WebLoader warning. To customize extraction from a URL, fetch manually and parse yourself:

Python · 12 lines
import requests
from zerda.document_loaders import HTMLStringLoader, HTMLLoaderConfig

response = requests.get("https://example.com/article", timeout=10)
response.raise_for_status()

config = HTMLLoaderConfig(
    tags_to_extract=["article"],
    extract_links=True,
    extract_tables=False,
)
docs = HTMLStringLoader(response.text, source=response.url, config=config).load()

Recipe: Crawling a rate-limited internal API/wiki gently

Python · 16 lines
from zerda.document_loaders import MultiURLLoader, WebLoaderConfig

# Conservative settings for a rate-sensitive target
config = WebLoaderConfig(
    timeout=20,
    max_retries=2,
    retry_delay=3.0,
    headers={"User-Agent": "MyCompanyBot/1.0 (contact: team@example.com)"},
)

docs = MultiURLLoader(
    url_list,
    config=config,
    max_workers=2,               # low concurrency
    delay_between_requests=2.0,  # 2s minimum between request starts
).load()

Recipe: Error visibility on large batches

MultiURLLoader with continue_on_error=True (the default) never raises for individual URL failures — failures only appear in logs. Enable logging to see them:

Python · 7 lines
import logging
logging.basicConfig(level=logging.INFO)

docs = MultiURLLoader(large_url_list, continue_on_error=True).load()
# Watch for:
#   ✗ Failed: {url} - {error}
#   "Failed to load N/M URLs. Failed: [...]"

If you need programmatic access to which URLs failed (not just log lines), wrap the loop yourself instead of using MultiURLLoader's built-in aggregation:

Python · 13 lines
from zerda.document_loaders import WebLoader

def fetch_with_tracking(urls, config=None):
    succeeded, failed = [], []
    for url in urls:
        try:
            succeeded.extend(WebLoader(url, config=config).load())
        except Exception as e:
            failed.append((url, str(e)))
    return succeeded, failed

docs, failures = fetch_with_tracking(url_list)
print(f"{len(failures)} URLs failed:", failures)

Recipe: Strict mode — abort the whole batch on first failure

Python · 5 lines
try:
    docs = MultiURLLoader(url_list, continue_on_error=False).load()
except Exception as e:
    print(f"Aborted: {e}")
    # No documents from any URL are usable here — handle upstream retry/alerting

SSRF caution for user-submitted URLs

If any of these URLs come from untrusted user input (rather than a fixed, developer-controlled list), read Security → No URL allowlisting or SSRF protection before deploying this pattern — WebLoader/MultiURLLoader perform no host validation of their own.

Related pages