ZerdaZerda

SensitiveDataMasker

On this page

Purpose

Scans free-form text (or nested dicts/lists) for common PII/secret patterns and replaces matches with placeholder tokens like [EMAIL] or [SSN], before that content is stored or sent to an LLM. This is the class AIMemoryManager uses (when enable_privacy_masking=True, the default) to mask user_input/assistant_output before they enter any memory tier.

Constructor

Python · 5 lines
def __init__(
    self,
    custom_patterns: Optional[List[Tuple[str, str]]] = None,
    placeholder_fmt: str = "[{label}]",
) -> None
ParameterTypeRequiredDefaultNotes
custom_patternsList[Tuple[str, str]] (label, regex string)NoNoneAppended after the 10 built-ins, compiled with re.I (case-insensitive) regardless of what case-sensitivity you intended.
placeholder_fmtstrNo"[{label}]"Must contain a {label} placeholder if overridden.

Public methods

mask(text: str) -> Tuple[str, Dict[str, int]]

Returns (masked_text, report) where report maps each pattern label that matched at least once to its match count. Labels with zero matches are omitted from report entirely (not present as 0).

mask_entry_content(content: Any) -> Tuple[Any, Dict[str, int]]

Recursively masks strings inside arbitrarily nested dict/list structures, merging per-label counts across the whole structure. Non-str/dict/list values (e.g. int, float, None) are returned unchanged with an empty report — not stringified and scanned.

has_sensitive_data(text: str) -> bool

any(pattern.search(text) for _, pattern in self.patterns) — a fast boolean check without performing any replacement, useful for pre-flight checks before deciding whether to log/store raw text at all.

Raises

re.error if a custom_patterns regex string is invalid — propagates from re.compile() inside __init__.

Thread safety

Read-only after construction (the compiled pattern list never mutates post-__init__), so calling mask/mask_entry_content/has_sensitive_data concurrently from multiple threads on the same instance is safe — each call only reads self.patterns and operates on its own local text/report variables.

Complexity

O(n × p) per call, where n is text length and p is the number of patterns (10 + any custom ones) — each pattern does a full-text findall/sub pass.

Examples

Python · 31 lines
from zerda.memory import SensitiveDataMasker

masker = SensitiveDataMasker()

# 1. Basic
clean, report = masker.mask("Contact me at jane@example.com or 555-123-4567")
print(clean)   # "Contact me at [EMAIL] or [PHONE_INTL]"
print(report)  # {"EMAIL": 1, "PHONE_INTL": 1}

# 2. Intermediate — nested structure
content = {"input": "My SSN is 123-45-6789", "meta": {"note": "call 192.168.1.1"}}
cleaned, report = masker.mask_entry_content(content)
print(cleaned)
# {"input": "My SSN is [SSN]", "meta": {"note": "call [IP_V4]"}}

# 3. Advanced — custom pattern for an internal ID format
masker2 = SensitiveDataMasker(custom_patterns=[("EMPLOYEE_ID", r"EMP-\d{6}")])
clean2, _ = masker2.mask("Assigned to EMP-004821")
print(clean2)  # "Assigned to [EMPLOYEE_ID]"

# 4. Error handling — invalid custom regex
try:
    SensitiveDataMasker(custom_patterns=[("BAD", r"(unclosed[")])
except Exception as e:
    print(type(e).__name__, e)  # re.error

# 5. Production — pre-flight check before logging raw user input
if masker.has_sensitive_data(raw_user_input):
    logger.warning("Sensitive data detected in input, masking before log")
    raw_user_input, _ = masker.mask(raw_user_input)
logger.info(raw_user_input)

Best practices

  • Run has_sensitive_data() before any raw-text logging path, separate from the masking used for LLM prompts/storage.
  • Add domain-specific custom_patterns for identifiers unique to your application (employee IDs, internal ticket numbers, etc.) rather than relying only on the generic built-ins.
  • Treat the built-in CREDIT_CARD/IBAN/PHONE_INTL patterns as heuristics — validate against representative samples of your own traffic for false-positive rate before depending on them for compliance.

Common mistakes

  • Assuming custom_patterns respects the case-sensitivity implied by how you wrote the regex — all patterns (built-in and custom) are compiled with re.I.
  • Passing non-string leaf values into mask_entry_content and expecting them scanned — numbers/None/etc. pass through untouched.

Related pages