WebLoader & MultiURLLoader

Module: web_loader · Extends BaseDocumentLoader directly

WebLoader

Overview

Fetches a single URL over HTTP(S) and parses it as HTML, with retry/backoff handling. Internally builds on HTMLStringLoader for the actual parsing — WebLoader's own responsibility is exclusively the network fetch.

Signature

class WebLoader(BaseDocumentLoader):
    def __init__(
        self,
        url: str,
        config: Optional[WebLoaderConfig] = None,
        headers: Optional[Dict[str, str]] = None,
        timeout: int = 10,
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Parameters

Returns

List[LoadedDocument] — whatever HTMLStringLoader produces for the fetched page (main text + table documents), with url and source metadata fields set to the requested URL.

Raises

• ImportError — "requests is required..." if the requests package isn't installed
• ConnectionError — after exhausting config.max_retries attempts, wrapping the last requests.exceptions.RequestException
• Any exception from HTMLStringLoader/BeautifulSoup parsing (e.g. ImportError if bs4 is missing) propagates unchanged

Internal workflow — _fetch_url()

for attempt in range(self.config.max_retries):
    try:
        response = requests.get(url, headers=headers, timeout=..., verify=...)
        response.raise_for_status()
        response.encoding = config.encoding or response.apparent_encoding or "utf-8"
        return response.text
    except requests.exceptions.RequestException as e:
        if attempt < max_retries - 1:
            time.sleep(retry_delay * (2 ** attempt))   # exponential backoff
raise ConnectionError(...)

Algorithm: standard retry-with-exponential-backoff. retry_delay * 2**attempt — e.g. with the default retry_delay=1.0, delays are 1s, 2s, 4s for attempts 0, 1, 2 (the loop runs max_retries total attempts, sleeping between all but the last). raise_for_status() means any 4xx/5xx response is treated as a retryable failure, not silently accepted.

load() then builds a fixed HTMLLoaderConfig (not the user's WebLoaderConfig) for the HTML parsing step:

html_config = HTMLLoaderConfig(
    tags_to_remove=["script", "style", "nav", "footer", "header", "aside"],
    extract_links=False,
    extract_tables=True,
)

[!WARNING] Configuration scope This means HTMLLoaderConfig customization (e.g. tags_to_extract, extract_links) is not exposed through WebLoader's public API — the HTML parsing behavior is hardcoded as shown above. If you need custom HTML extraction from a web page, fetch the HTML yourself (e.g. with requests) and pass it to HTMLStringLoader directly with your own HTMLLoaderConfig.

Side effects

• Makes outbound HTTP requests (up to max_retries per call).
• time.sleep() between retries blocks the calling thread.
• No caching — every load() call re-fetches the URL.

Thread safety

A single WebLoader instance has no shared mutable state across load() calls and is safe to call from multiple threads, though doing so against the same URL repeatedly is wasteful (no caching). For fetching many URLs concurrently, use MultiURLLoader rather than manually threading WebLoader instances.

Examples

Basic:

from zerda.document_loaders import WebLoader

docs = WebLoader("https://example.com/article").load()
print(docs[0].metadata["title"])

Custom headers and timeout:

docs = WebLoader(
    "https://api.example.com/page",
    headers={"Authorization": "Bearer token123"},
    timeout=30,
).load()

Custom retry policy:

from zerda.document_loaders import WebLoader, WebLoaderConfig

config = WebLoaderConfig(max_retries=5, retry_delay=2.0, verify_ssl=False)
docs = WebLoader("https://flaky-site.example.com", config=config).load()

Error handling:

try:
    docs = WebLoader("https://unreachable.example.com").load()
except ConnectionError as e:
    print(f"Failed after retries: {e}")

Production usage — defensive fetch with logging:

import logging

def safe_fetch(url: str):
    try:
        return WebLoader(url, config=WebLoaderConfig(max_retries=2, timeout=8)).load()
    except (ConnectionError, ImportError) as e:
        logging.error(f"Failed to load {url}: {e}")
        return []

Best practices

• Set verify_ssl=False only for trusted internal/dev environments — disabling certificate verification is a security risk in production.
• Tune timeout per source: slow third-party sites may need a higher value than the 10s default to avoid premature failures counted against max_retries.

Common mistakes

• Expecting WebLoaderConfig.extract_metadata to control whether title/meta tags are extracted — this field exists in the dataclass but is never read by web_loader.py; metadata extraction is governed by the hardcoded HTMLLoaderConfig shown above, which always behaves like HTMLLoaderConfig()'s defaults for metadata purposes.
• Passing a WebLoaderConfig and expecting its headers to be the only headers sent — they're merged with (and the default User-Agent is always added unless overridden by the same key in your headers dict).

MultiURLLoader

Overview

Fetches multiple URLs concurrently using a thread pool, with global rate limiting and per-URL error isolation.

Signature

class MultiURLLoader(BaseDocumentLoader):
    def __init__(
        self,
        urls: List[str],
        config: Optional[WebLoaderConfig] = None,
        max_workers: int = 4,
        delay_between_requests: float = 0.5,
        continue_on_error: bool = True,
    ): ...

    def load(self) -> List[LoadedDocument]: ...

Parameters

Returns

List[LoadedDocument] — concatenation of every successful URL's documents, in completion order (not input order, since as_completed() is used).

Raises

• Propagates the first exception encountered if and only if continue_on_error=False — otherwise all errors are caught, logged, and collected internally; load() itself does not raise for individual URL failures.

Internal workflow — concurrency & rate limiting

sem = threading.Semaphore(1)
last_time = {"t": 0.0}

def _load_with_throttle(url):
    with sem:
        elapsed = time.monotonic() - last_time["t"]
        if elapsed < delay:
            time.sleep(delay - elapsed)
        last_time["t"] = time.monotonic()
    # fetch happens OUTSIDE the semaphore
    return WebLoader(url=url, config=config).load()

with ThreadPoolExecutor(max_workers=max_workers) as executor:
    futures = {executor.submit(_load_with_throttle, url): url for url in urls}
    for future in as_completed(futures):
        ...

Algorithm — rate limiting design: a Semaphore(1) (effectively a mutex) plus a shared last_time dict enforce that request starts are spaced at least delay_between_requests seconds apart, globally across all worker threads — this prevents bursting many simultaneous requests at a rate-limited server. Critically, the semaphore is released before the actual WebLoader(...).load() call, so the network I/O and retry/backoff logic of multiple requests can still overlap in time even though their start times are throttled — this maximizes throughput while still respecting a minimum inter-request spacing.

For each completed future, on success: all_docs.extend(docs) and an info-level log. On failure: appended to an errors list and logged at error level; if continue_on_error=False, the exception is immediately re-raised (which will also cause ThreadPoolExecutor's context manager to wait for other in-flight futures before propagating, per standard concurrent.futures semantics).

Metadata produced

Same as WebLoader per successful URL — no additional batch-level metadata is added to individual documents (errors are only visible via logs, not returned alongside successful results).

Side effects

• Up to max_workers concurrent outbound HTTP connections.
• Logs ✓ Loaded: {url} ({n} docs) per success and ✗ Failed: {url} - {error} per failure, plus a summary warning if any URLs failed.

Complexity

• Time: wall-clock roughly max(per-URL fetch times) + (n_urls / max_workers) * delay_between_requests in the worst case, bounded below by the rate limit and above by sequential fetching.
• Space: O(total documents across all successful URLs) — all results are accumulated before load() returns; there's no streaming/lazy variant for multi-URL loading (MultiURLLoader does not override lazy_load()).

Thread safety

This is the package's primary concurrency-oriented loader. Internally thread-safe by design (semaphore-protected shared state); each load() call is self-contained.

Examples

Basic:

from zerda.document_loaders import MultiURLLoader

urls = [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3",
]
docs = MultiURLLoader(urls).load()
print(f"Loaded {len(docs)} documents from {len(urls)} URLs")

Higher concurrency, faster rate:

docs = MultiURLLoader(urls, max_workers=10, delay_between_requests=0.1).load()

Strict mode — abort on first failure:

try:
    docs = MultiURLLoader(urls, continue_on_error=False).load()
except Exception as e:
    print(f"Batch aborted: {e}")

Polite crawling of a single rate-limited domain:

from zerda.document_loaders import MultiURLLoader, WebLoaderConfig

docs = MultiURLLoader(
    urls,
    max_workers=2,
    delay_between_requests=2.0,
    config=WebLoaderConfig(timeout=15, max_retries=2),
).load()

Production usage — large URL batch with error visibility:

import logging
logging.basicConfig(level=logging.INFO)

loader = MultiURLLoader(url_list, max_workers=8, continue_on_error=True)
docs = loader.load()
# Check logs for "Failed to load N/M URLs. Failed: [...]"

Best practices

• Set delay_between_requests based on the target site's rate-limit policy, not arbitrarily low — being a polite crawler matters for both ethics and avoiding IP bans.
• Use continue_on_error=True (default) for large batches where a few failed URLs shouldn't block the rest; switch to False only when every URL is expected to succeed and any failure should halt processing immediately.

Common mistakes

• Assuming results preserve input urls order — they don't; as_completed() yields futures in completion order, which is generally non-deterministic under concurrency.
• Setting max_workers very high without lowering delay_between_requests accordingly and being surprised requests still queue up — the semaphore enforces a hard floor on the global request start rate regardless of worker count.

Related APIs

• WebLoaderConfig
• HTMLStringLoader — used internally by WebLoader for parsing
• AutoLoader — dispatches http(s):///ftp:// URLs to WebLoader automatically
• LoadedDocument