feat!: optional fail-closed fallback, legend output, richer prompt & docs

BREAKING CHANGE: regex fallback is now opt-in (patterns no longer defaults to presets.swiss). With no fallback an LLM failure throws AnonymizationError (fail-closed) and the pre-filter is bypassed; at least one of llm/patterns is required. AnonymizationResult gains a required `legend`; anonymizeChunks seed is now { mapping, legend? } and returns legend. - prompt: model may coin new UPPERCASE abbreviations and returns a 'legende' explaining every abbreviation used (French); backfilled by DEFAULT_LEGEND - PatternDef.meaning surfaces in the legend; swiss/generic presets get meanings - AnonymizationError (exported) wraps the cause on fail-closed - README: drop the chatbot provenance line; add 'How it works' + nLPD sections - 34 tests / 99% coverage; bump to 0.2.0 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-25 15:28:04 +01:00
parent 08f84ae14a
commit 669794522e
13 changed files with 408 additions and 83 deletions
--- a/README.md
+++ b/README.md
@@ -11,16 +11,18 @@ It replaces personal data in text with stable placeholders before the text leave
 values afterwards — including across **streamed** tokens.

 - 🔌 **Pluggable LLM detection** — catch free-form PII (names, addresses) via any OpenAI-compatible
-  endpoint, or your own provider. No LLM? It degrades gracefully to regex.
- 🧩 **Deterministic regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
-  pattern presets (`swiss`, `generic`) or your own.
+  endpoint, or your own provider.
+- 🧩 **Optional regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
+  presets (`swiss`, `generic`) or your own. Opt in for graceful degradation, or omit it to **fail closed**.
+- 🏷️ **Self-describing tokens** — every result ships a `legend` (`PER`→`Personne`, `M`→`Masculin`) you can
+  hand to the downstream LLM so it understands the placeholders; the model may coin new abbreviations too.
 - 🔁 **Deterministic coreference** — the same person keeps the same id (`[PER_1]`) across a question and
  every retrieved chunk, with automatic de-collision.
 - 🌊 **Streaming-safe** — a placeholder split across two stream chunks (`[PER_` + `1.NOM:M]`) is never
  leaked partially.
 - 🪶 **Zero runtime dependencies**, ESM + CJS, fully typed.

-> Built by [Mobiletic](https://mobiletic.com) and extracted from a production Swiss-nLPD chatbot.
+> Built by [Mobiletic](https://mobiletic.com).

 ## Install

@@ -39,9 +41,10 @@ import { Anonymizer, presets } from '@mobiletic/anonymizer';

 const anonymizer = new Anonymizer({ patterns: presets.swiss });

-const { anon, mapping } = await anonymizer.anonymize('Écris à jean@exemple.ch');
+const { anon, mapping, legend } = await anonymizer.anonymize('Écris à jean@exemple.ch');
 // anon    -> "Écris à [EMAIL_1]"
-// mapping -> { "[EMAIL_1]": "jean@exemple.ch" }
+// mapping -> { "[EMAIL_1]": "jean@exemple.ch" }   (secret — keep on your side)
+// legend  -> { "EMAIL": "Adresse e-mail" }        (safe to share downstream)

 anonymizer.deanonymize(anon, mapping); // -> "Écris à jean@exemple.ch"
 ```
@@ -58,15 +61,19 @@ const anonymizer = new Anonymizer({
    model: process.env.LLM_MODEL!,
    timeoutMs: 3000,
  }),
-  patterns: presets.swiss, // regex fallback if the LLM is down/misbehaves
+  patterns: presets.swiss, // OPTIONAL regex fallback if the LLM is down/misbehaves
 });

-const { anon, mapping } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
-// anon -> "Le dossier de [PER_1.NOM:M] est complet."
+const { anon, mapping, legend } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
+// anon   -> "Le dossier de [PER_1.NOM:M] est complet."
+// legend -> { "PER": "Personne", "NOM": "Nom de famille", "M": "Masculin" }
 ```

-If the LLM call fails, times out, or returns an invalid shape, the anonymizer automatically falls back to
-the regex engine — it never throws on a provider failure.
+**Fallback is opt-in (fail-closed).** If a `patterns` fallback is configured, a failed/timed-out/invalid
+LLM call degrades to the regex engine. If you omit `patterns`, there's nothing to degrade to, so the call
+**throws an `AnonymizationError`** (with the underlying error as `.cause`) — it never silently returns
+un-anonymized text. With no fallback the pre-filter is also bypassed, so every non-trivial call consults
+the LLM (more calls, no leaks). At least one of `llm` or `patterns` is required.

 `openAICompatibleProvider` retries **transient** failures (network error, timeout, HTTP 429/5xx) before
 giving up; 4xx and malformed responses are not retried. Tune with `timeoutMs` (per attempt, default 3000),
@@ -86,13 +93,14 @@ process.stdout.write(stream.flush());

 ### Anonymizing retrieved chunks consistently (RAG)

-`anonymizeChunks(chunks, seed)` reuses the question's mapping (`seed`) so the same person gets the same id
-across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:
+`anonymizeChunks(chunks, seed)` reuses the question's `{ mapping, legend }` so the same person gets the
+same id across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:

 ```ts
 const q = await anonymizer.anonymize(question);
-const { anon, mapping } = await anonymizer.anonymizeChunks(retrievedChunks, q.mapping);
-// `mapping` is the full question ∪ chunks table; pass it to deanonymize()/makeStreamDeanonymizer().
+const { anon, mapping, legend } = await anonymizer.anonymizeChunks(retrievedChunks, q);
+// `mapping`/`legend` are the full question ∪ chunks tables;
+// pass `mapping` to deanonymize()/makeStreamDeanonymizer(), and `legend` to the downstream LLM.
 ```

 ## Configuration
@@ -100,10 +108,11 @@ const { anon, mapping } = await anonymizer.anonymizeChunks(retrievedChunks, q.ma
 ```ts
 new Anonymizer({
  llm?,       // LlmProvider — omit for regex-only mode
-  patterns?,  // PatternDef[] — defaults to presets.swiss
+  patterns?,  // PatternDef[] — opt-in regex fallback; omit to fail closed
  nameHint?,  // RegExp flagging likely names so the LLM is consulted (has a default)
  logger?,    // { warn(msg) } — receives fallback warnings; defaults to no-op
 });
+// At least one of `llm` or `patterns` must be provided, or the constructor throws.
 ```

 ### Presets & custom patterns
@@ -142,10 +151,10 @@ import type { LlmProvider } from '@mobiletic/anonymizer';
 const myProvider: LlmProvider = {
  isConfigured: () => true,
  async anonymize(text) {
-    /* return { anon, mapping } */
+    /* return { anon, mapping, legend } */
  },
  async anonymizeBatch(texts, usedIds) {
-    /* return { segments, mapping } */
+    /* return { segments, mapping, legend } */
  },
 };
 ```
@@ -157,7 +166,69 @@ const myProvider: LlmProvider = {
 [EMAIL_1]       structured id from the regex fallback
 ```

-`PLACEHOLDER_RE` is exported if you need to scan text for placeholders.
+`PLACEHOLDER_RE` is exported if you need to scan text for placeholders. The LLM may also **coin new
+abbreviations** (always uppercase `[A-Z_]`) for entities/attributes/context it discovers — every one it
+uses is described in the result `legend`.
+
+## How it works
+
+The library does **query-time pseudonymization**: it rewrites text so that personal data never leaves your
+trust boundary in identifiable form, while keeping the answer fully reversible on your side.
+
+```
+                        ┌───────────────────────────── your trust boundary ──────────────────────────────┐
+  raw text  ─▶  ① pre-filter ─▶ ② LLM detect ─▶ ③ regex fallback ─▶ ④ validate ─▶  anon + mapping + legend
+                  (skip if            (names,         (optional;          (every                │       │
+                   clearly            addresses,      structured ids;     placeholder    ┌───────┘       │
+                   PII-free)          coins abbrevs,  fail closed if      is mapped)      ▼               ▼
+                                      builds legend)  absent)                          mapping         legend
+                                                                                     (SECRET,        (shareable:
+   downstream LLM  ◀── anon + legend ─────────────────────────────────────────────  reversible)     PER=Personne…)
+   answer (with [PER_1.NOM:M] tokens)
+        │
+        ▼
+  ⑤ stream de-anonymize ──▶ real values restored for the end user (placeholders never leak, even if split)
+```
+
+1. **Pre-filter** — a cheap regex/name check skips the LLM round-trip for text that clearly has no PII.
+   (Bypassed when no regex fallback is configured, so nothing slips through.)
+2. **LLM detection** — finds free-form PII a regex can't (names, addresses), keeps **coreference** (the
+   same person is always `[PER_1]`), coins uppercase abbreviations for anything new, and returns a
+   **legend** describing them.
+3. **Regex fallback** — optional, deterministic detection of structured identifiers; used if the LLM is
+   unavailable. Omit it to **fail closed** (raise `AnonymizationError` rather than risk a leak).
+4. **Validation** — bidirectional check that every placeholder has a mapping entry and vice-versa.
+5. **Streaming de-anonymization** — `makeStreamDeanonymizer` restores real values token-by-token, buffering
+   any placeholder split across chunks so a partial `[PER_` is never emitted.
+
+Three distinct outputs, with different sensitivities:
+
+| Output    | Example                                  | Sensitivity                                             |
+| --------- | ---------------------------------------- | ------------------------------------------------------- |
+| `anon`    | `Le dossier de [PER_1.NOM:M]`            | Safe to send onward — no identifiers                    |
+| `mapping` | `{ "[PER_1.NOM:M]": "Alain Jaccard" }`   | **Secret** — re-identifies people; keep it on your side |
+| `legend`  | `{ "PER": "Personne", "M": "Masculin" }` | Safe to share — explains the tokens to a downstream LLM |
+
+## How this helps with the nLPD
+
+Switzerland's [nLPD](https://www.fedlex.admin.ch/eli/cc/2022/491/fr) (and the GDPR) push for **data
+minimisation** and favour **pseudonymisation** when personal data is processed by third parties. This
+library is built around those principles:
+
+- **The third party never sees raw PII.** When you send text to an external LLM (or any external service),
+  it receives only pseudonymised tokens like `[PER_1.NOM:M]` plus the non-identifying `legend` — never the
+  real name, e-mail, AVS number, etc.
+- **Pseudonymisation, not loss of meaning.** The re-identification key (`mapping`) stays in your
+  infrastructure; only you can reverse the tokens. The `legend` lets the downstream model still reason
+  correctly ("a person", "male") without knowing _who_.
+- **Fail-closed option.** Omitting the regex fallback means that if detection can't run, the call errors
+  instead of forwarding data that wasn't pseudonymised — no silent leak.
+- **Coreference & minimisation.** Re-using one id per entity avoids spreading extra distinguishing detail
+  across a prompt, and the corpus itself can stay in clear text — pseudonymisation happens only at the
+  boundary, at query time.
+
+> This is an engineering aid, not legal advice or a certification. You remain the data controller; assess
+> it against your own obligations (see the disclaimer below).

 ## Compliance note