41d8993137
Each example now shows the shareable `legend` as a table (Abbreviation → meaning), marking 🆕 the abbreviations the model coined for attributes absent from the base prompt (ASSUR_ID, PERMIS, PASSPORT, RCC, BIO_ID, SANG, DOSSIER…). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
350 lines
19 KiB
Markdown
350 lines
19 KiB
Markdown
# @mobiletic/anonymizer
|
||
|
||
[](https://git.mobiletic.net/mobiletic/anonymizer/actions)
|
||
[](https://www.npmjs.com/package/@mobiletic/anonymizer)
|
||
[](./LICENSE)
|
||
|
||
Framework-agnostic **PII anonymization & pseudonymization** for TypeScript/JavaScript.
|
||
|
||
It replaces personal data in text with stable placeholders before the text leaves your trust boundary
|
||
(e.g. before sending it to a third-party LLM, log sink, or analytics pipeline), and restores the real
|
||
values afterwards — including across **streamed** tokens.
|
||
|
||
- 🔌 **Pluggable LLM detection** — catch free-form PII (names, addresses) via any OpenAI-compatible
|
||
endpoint, or your own provider.
|
||
- 🧩 **Optional regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
|
||
presets (`swiss`, `generic`) or your own. Opt in for graceful degradation, or omit it to **fail closed**.
|
||
- 🏷️ **Self-describing tokens** — every result ships a `legend` (`PER`→`Personne`, `M`→`Masculin`) you can
|
||
hand to the downstream LLM so it understands the placeholders; the model may coin new abbreviations too.
|
||
- 🔁 **Deterministic coreference** — the same person keeps the same id (`[PER_1]`) across a question and
|
||
every retrieved chunk, with automatic de-collision.
|
||
- 🌊 **Streaming-safe** — a placeholder split across two stream chunks (`[PER_` + `1.NOM:M]`) is never
|
||
leaked partially.
|
||
- 🪶 **Zero runtime dependencies**, ESM + CJS, fully typed.
|
||
|
||
> Built by [Mobiletic](https://mobiletic.com).
|
||
|
||
## Install
|
||
|
||
```bash
|
||
npm install @mobiletic/anonymizer
|
||
```
|
||
|
||
Requires Node ≥ 18 (uses native `fetch`).
|
||
|
||
## Quick start
|
||
|
||
### Regex-only (no LLM, fully deterministic)
|
||
|
||
```ts
|
||
import { Anonymizer, presets } from '@mobiletic/anonymizer';
|
||
|
||
const anonymizer = new Anonymizer({ patterns: presets.swiss });
|
||
|
||
const { anon, mapping, legend } = await anonymizer.anonymize('Écris à jean@exemple.ch');
|
||
// anon -> "Écris à [EMAIL_1]"
|
||
// mapping -> { "[EMAIL_1]": "jean@exemple.ch" } (secret — keep on your side)
|
||
// legend -> { "EMAIL": "Adresse e-mail" } (safe to share downstream)
|
||
|
||
anonymizer.deanonymize(anon, mapping); // -> "Écris à jean@exemple.ch"
|
||
```
|
||
|
||
### With an LLM (also catches names, addresses…)
|
||
|
||
```ts
|
||
import { Anonymizer, openAICompatibleProvider, presets } from '@mobiletic/anonymizer';
|
||
|
||
const anonymizer = new Anonymizer({
|
||
llm: openAICompatibleProvider({
|
||
baseUrl: process.env.LLM_BASE_URL!, // OpenAI, Infomaniak, vLLM, Ollama, …
|
||
apiKey: process.env.LLM_API_KEY!,
|
||
model: process.env.LLM_MODEL!,
|
||
timeoutMs: 3000,
|
||
}),
|
||
patterns: presets.swiss, // OPTIONAL regex fallback if the LLM is down/misbehaves
|
||
});
|
||
|
||
const { anon, mapping, legend } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
|
||
// anon -> "Le dossier de [PER_1.NOM:M] est complet."
|
||
// legend -> { "PER": "Personne", "NOM": "Nom de famille", "M": "Masculin" }
|
||
```
|
||
|
||
**Fallback is opt-in (fail-closed).** If a `patterns` fallback is configured, a failed/timed-out/invalid
|
||
LLM call degrades to the regex engine. If you omit `patterns`, there's nothing to degrade to, so the call
|
||
**throws an `AnonymizationError`** (with the underlying error as `.cause`) — it never silently returns
|
||
un-anonymized text. With no fallback the pre-filter is also bypassed, so every non-trivial call consults
|
||
the LLM (more calls, no leaks). At least one of `llm` or `patterns` is required.
|
||
|
||
`openAICompatibleProvider` retries **transient** failures (network error, timeout, HTTP 429/5xx) before
|
||
giving up; 4xx and malformed responses are not retried. Tune with `timeoutMs` (per attempt, default 3000),
|
||
`retries` (default 1 → 2 attempts), and `retryDelayMs` (linear backoff, default 250). Worst-case latency
|
||
is `(retries + 1) × timeoutMs`, so keep `retries` low on latency-sensitive paths.
|
||
|
||
It works with **Infomaniak** and other open-model endpoints out of the box: `response_format` is **omitted
|
||
by default** (Infomaniak rejects the legacy `{ type: 'json_object' }`), and responses are parsed leniently
|
||
(a fenced JSON code block or surrounding prose is tolerated). For endpoints that support it, opt in with
|
||
`responseFormat` — e.g. `{ type: 'json_object' }` or a `json_schema` object.
|
||
|
||
### Streaming de-anonymization
|
||
|
||
When you stream an LLM answer back to a user, restore real values without ever emitting a half-written
|
||
placeholder:
|
||
|
||
```ts
|
||
const stream = anonymizer.makeStreamDeanonymizer(mapping);
|
||
for await (const token of llmTokens) process.stdout.write(stream.push(token));
|
||
process.stdout.write(stream.flush());
|
||
```
|
||
|
||
### Anonymizing retrieved chunks consistently (RAG)
|
||
|
||
`anonymizeChunks(chunks, seed)` reuses the question's `{ mapping, legend }` so the same person gets the
|
||
same id across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:
|
||
|
||
```ts
|
||
const q = await anonymizer.anonymize(question);
|
||
const { anon, mapping, legend } = await anonymizer.anonymizeChunks(retrievedChunks, q);
|
||
// `mapping`/`legend` are the full question ∪ chunks tables;
|
||
// pass `mapping` to deanonymize()/makeStreamDeanonymizer(), and `legend` to the downstream LLM.
|
||
```
|
||
|
||
### Multi-turn conversations
|
||
|
||
In a chat, anonymizing each message independently would renumber entities every turn (Adil could become
|
||
`PER_1` in turn 2 while Oussama was `PER_1` in turn 1). `anonymizeTurn` threads a serializable
|
||
`AnonymizerSession` so **one entity keeps one id across the whole conversation**:
|
||
|
||
```ts
|
||
let session; // persist this per conversation (Redis/DB); pass it back each turn
|
||
for (const message of userTurns) {
|
||
const turn = await anonymizer.anonymizeTurn(message, session);
|
||
session = turn.session; // { mapping, legend, history } — carries coreference forward
|
||
send(turn.anon, turn.legend); // → the downstream chatbot (placeholders only)
|
||
}
|
||
// restore the bot's placeholder-bearing reply for the user:
|
||
anonymizer.deanonymize(botReply, session.mapping);
|
||
```
|
||
|
||
Or the stateful wrapper for in-memory use:
|
||
|
||
```ts
|
||
const conv = anonymizer.conversation();
|
||
await conv.anonymize('Bonjour, je suis Oussama'); // → Oussama = [PER_1…]
|
||
await conv.anonymize('Mon amie Adil …'); // → Oussama stays [PER_1…], Adil = [PER_2…]
|
||
conv.deanonymize(botReply);
|
||
```
|
||
|
||
Under the hood: known values are re-substituted locally (`applyKnown`), the provider is told which ids are
|
||
taken, and new entities are de-collided into the session — so numbering never drifts.
|
||
|
||
**Trust boundary & the `mapping`.** The real boundary is "keep PII out of the _downstream chatbot_" — it
|
||
only ever receives placeholders + `legend`. Your **anonymizer** endpoint already sees the cleartext it's
|
||
asked to anonymize, so if (and only if) it's a _trusted_ processor you may give it richer context — set
|
||
`includeMappingInContext: true` on `openAICompatibleProvider` to also send the `mapping` for maximum
|
||
cross-turn accuracy. It's **off by default**; keep it off for third-party endpoints you don't trust with
|
||
raw values.
|
||
|
||
## Configuration
|
||
|
||
```ts
|
||
new Anonymizer({
|
||
llm?, // LlmProvider — omit for regex-only mode
|
||
patterns?, // PatternDef[] — opt-in regex fallback; omit to fail closed
|
||
nameHint?, // RegExp flagging likely names so the LLM is consulted (has a default)
|
||
logger?, // { warn(msg) } — receives fallback warnings; defaults to no-op
|
||
});
|
||
// At least one of `llm` or `patterns` must be provided, or the constructor throws.
|
||
```
|
||
|
||
### Presets & custom patterns
|
||
|
||
```ts
|
||
import { presets } from '@mobiletic/anonymizer';
|
||
|
||
presets.swiss; // AVS, IBAN CH, EMAIL, Swiss phone, DATE
|
||
presets.generic; // EMAIL, IBAN, credit card, IPv4, phone, DATE
|
||
|
||
// Compose / extend:
|
||
const patterns = [
|
||
...presets.generic,
|
||
{ tag: 'TICKET', re: /\bJIRA-\d+\b/g }, // patterns must use the global flag
|
||
];
|
||
```
|
||
|
||
Each pattern may carry an optional `validate(match) => boolean` second stage — a match is only redacted
|
||
if it passes. `presets.generic` uses it for a [Luhn](https://en.wikipedia.org/wiki/Luhn_algorithm) check
|
||
so arbitrary long digit runs aren't mistaken for credit cards:
|
||
|
||
```ts
|
||
{ tag: 'CREDIT_CARD', re: /\b\d(?:[ -]?\d){12,18}\b/g, validate: luhnValid }
|
||
```
|
||
|
||
> The `generic` preset is a **best-effort** starting point — broad patterns (phone, date, card) can
|
||
> overlap. For production use, prefer a locale-specific preset (`presets.swiss`) or your own patterns.
|
||
|
||
### Custom LLM provider
|
||
|
||
Implement `LlmProvider` to use any backend (Anthropic, a local model, a rules engine…):
|
||
|
||
```ts
|
||
import type { LlmProvider } from '@mobiletic/anonymizer';
|
||
|
||
const myProvider: LlmProvider = {
|
||
isConfigured: () => true,
|
||
async anonymize(text) {
|
||
/* return { anon, mapping, legend } */
|
||
},
|
||
async anonymizeBatch(texts, usedIds) {
|
||
/* return { segments, mapping, legend } */
|
||
},
|
||
};
|
||
```
|
||
|
||
## Placeholder format
|
||
|
||
```
|
||
[PER_1.NOM:M] entity PER #1, attribute NOM, context M (rich, from the LLM)
|
||
[EMAIL_1] structured id from the regex fallback
|
||
```
|
||
|
||
`PLACEHOLDER_RE` is exported if you need to scan text for placeholders. The LLM may also **coin new
|
||
abbreviations** (always uppercase `[A-Z_]`) for entities/attributes/context it discovers — every one it
|
||
uses is described in the result `legend`.
|
||
|
||
## How it works
|
||
|
||
The library does **query-time pseudonymization**: it rewrites text so that personal data never leaves your
|
||
trust boundary in identifiable form, while keeping the answer fully reversible on your side.
|
||
|
||
```
|
||
┌───────────────────────────── your trust boundary ──────────────────────────────┐
|
||
raw text ─▶ ① pre-filter ─▶ ② LLM detect ─▶ ③ regex fallback ─▶ ④ validate ─▶ anon + mapping + legend
|
||
(skip if (names, (optional; (every │ │
|
||
clearly addresses, structured ids; placeholder ┌───────┘ │
|
||
PII-free) coins abbrevs, fail closed if is mapped) ▼ ▼
|
||
builds legend) absent) mapping legend
|
||
(SECRET, (shareable:
|
||
downstream LLM ◀── anon + legend ───────────────────────────────────────────── reversible) PER=Personne…)
|
||
answer (with [PER_1.NOM:M] tokens)
|
||
│
|
||
▼
|
||
⑤ stream de-anonymize ──▶ real values restored for the end user (placeholders never leak, even if split)
|
||
```
|
||
|
||
1. **Pre-filter** — a cheap regex/name check skips the LLM round-trip for text that clearly has no PII.
|
||
(Bypassed when no regex fallback is configured, so nothing slips through.)
|
||
2. **LLM detection** — finds free-form PII a regex can't (names, addresses), keeps **coreference** (the
|
||
same person is always `[PER_1]`), coins uppercase abbreviations for anything new, and returns a
|
||
**legend** describing them.
|
||
3. **Regex fallback** — optional, deterministic detection of structured identifiers; used if the LLM is
|
||
unavailable. Omit it to **fail closed** (raise `AnonymizationError` rather than risk a leak).
|
||
4. **Validation** — bidirectional check that every placeholder has a mapping entry and vice-versa.
|
||
5. **Streaming de-anonymization** — `makeStreamDeanonymizer` restores real values token-by-token, buffering
|
||
any placeholder split across chunks so a partial `[PER_` is never emitted.
|
||
|
||
Three distinct outputs, with different sensitivities:
|
||
|
||
| Output | Example | Sensitivity |
|
||
| --------- | ---------------------------------------- | ------------------------------------------------------- |
|
||
| `anon` | `Le dossier de [PER_1.NOM:M]` | Safe to send onward — no identifiers |
|
||
| `mapping` | `{ "[PER_1.NOM:M]": "Alain Jaccard" }` | **Secret** — re-identifies people; keep it on your side |
|
||
| `legend` | `{ "PER": "Personne", "M": "Masculin" }` | Safe to share — explains the tokens to a downstream LLM |
|
||
|
||
## Examples
|
||
|
||
Two examples produced **live by Gemma 4** on fictional Swiss e-learning data. `mapping` is the secret
|
||
re-identification key (kept by the operator); `legend` is safe to send downstream. The model even **coins
|
||
its own abbreviations** for attributes not in the base prompt (passport, permit, blood type, …).
|
||
|
||
[](./examples/RESULTS.md)
|
||
5 single messages of rising complexity **+** 5 multi-turn conversations, with full mappings & legends.
|
||
|
||
### A dense single message — 5 people + rare attributes
|
||
|
||
**User message:**
|
||
|
||
> Note de dossier complète : le patient mineur Noah Baumann (né le 12.06.2013, groupe sanguin B+, numéro d'assuré maladie 756.2211.9988.77, allergique aux arachides) … Ses parents, Delphine Rieder (mère, permis de séjour C, 021 555 12 34) et Marco Baumann (père, passeport italien YA9087654), cosignent … Le Dr Farah Haddad (n° RCC V123456) a ouvert le dossier médical DM-2025-0417 … l'agent Kevin Zbinden de la caisse Helvetia … Noah utilise l'identifiant biométrique BIO-7729.
|
||
|
||
**Anonymized — what the model sees** (5 distinct people `PER_1…PER_5` + `ORG_1`; `Noah` reused at the end):
|
||
|
||
> Note de dossier complète : le patient mineur [PER_1.PRENOM:U] [PER_1.NOM:U] (né le [PER_1.DATE_NAISSANCE:2013], groupe sanguin B+, numéro d'assuré maladie [PER_1.ASSUR_ID:U], allergique aux arachides) … Ses parents, [PER_2.PRENOM:F] [PER_2.NOM:F] (mère, permis de séjour [PER_2.PERMIS:C], [PER_2.TELEPHONE:U]) et [PER_3.PRENOM:M] [PER_3.NOM:M] (père, passeport italien [PER_3.PASSPORT:U]), cosignent … Le Dr [PER_4.PRENOM:F] [PER_4.NOM:F] (n° RCC [PER_4.RCC:U]) a ouvert le dossier médical DM-2025-0417 … l'agent [PER_5.PRENOM:M] [PER_5.NOM:M] de la caisse [ORG_1.NOM:Assurance] … [PER_1.PRENOM:U] utilise l'identifiant biométrique [PER_1.BIO_ID:U].
|
||
|
||
**`legend` 🏷️ (shareable with the downstream model; 🆕 = coined by the model, not in the base prompt):**
|
||
|
||
| Abbreviation | Meaning | |
|
||
| ---------------- | ------------------------- | --- |
|
||
| `PER` | Personne | |
|
||
| `PRENOM` / `NOM` | Prénom / Nom de famille | |
|
||
| `M` / `F` / `U` | Masculin / Féminin / n.d. | |
|
||
| `DATE_NAISSANCE` | Date de naissance | |
|
||
| `TELEPHONE` | Numéro de téléphone | |
|
||
| `ORG` | Organisation | |
|
||
| `ASSUR_ID` | Numéro d'assuré | 🆕 |
|
||
| `PERMIS` / `C` | Permis de séjour / type C | 🆕 |
|
||
| `PASSPORT` | Numéro de passeport | 🆕 |
|
||
| `RCC` | Numéro RCC (médecin) | 🆕 |
|
||
| `BIO_ID` | Identifiant biométrique | 🆕 |
|
||
|
||
### A multi-turn conversation — 4 people; blood group recalled turn 1 → turn 6
|
||
|
||
| Turn | User message | Anonymized (model sees) |
|
||
| ---- | ---------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
|
||
| 1 | …mon fils mineur, Noah Baumann, né le 12.06.2013, groupe sanguin B+. | …mon fils mineur, [PER_1.PRENOM:U] [PER_1.NOM:U], né le [PER_1.DATE_NAISSANCE:2013], groupe sanguin [PER_1.SANG:B+]. |
|
||
| 3 | Moi, sa mère Delphine Rieder … permis de séjour de type C. | Moi, sa mère [PER_2.PRENOM:F] [PER_2.NOM:F] … permis de séjour de type [PER_2.PERMIS:C]. |
|
||
| 4 | Son père, Marco Baumann, passeport italien YA9087654, … | Son père, [PER_3.PRENOM:M] [PER_1.NOM:U], passeport italien [PER_3.PASSPORT:ITA], … |
|
||
| 5 | Le Dr Farah Haddad a ouvert le dossier médical DM-2025-0417 pour Noah. | Le [PER_4.TITRE:MED] [PER_4.PRENOM:F] [PER_4.NOM:F] a ouvert le dossier médical [PER_1.DOSSIER:MED] pour [PER_1.PRENOM:U]. |
|
||
| 6 | Rappel : le groupe sanguin B+ de Noah doit figurer sur son badge… | Rappel : le groupe sanguin [PER_1.SANG:B+] de [PER_1.PRENOM:U] doit figurer… |
|
||
|
||
Noah stays `PER_1` across all six turns, and **his blood group `[PER_1.SANG:B+]` from turn 1 is reused in
|
||
turn 6** — long-range coreference holds. (The father shares the surname, so `[PER_1.NOM:U]` is reused for
|
||
it — the same value maps to the same token.)
|
||
|
||
**`legend` 🏷️ (cumulative across the conversation; 🆕 = coined by the model):**
|
||
|
||
| Abbreviation | Meaning | |
|
||
| ---------------- | ------------------------------ | --- |
|
||
| `PER` | Personne | |
|
||
| `PRENOM` / `NOM` | Prénom / Nom de famille | |
|
||
| `M` / `F` / `U` | Masculin / Féminin / inconnu | |
|
||
| `DATE_NAISSANCE` | Date de naissance | |
|
||
| `AVS` | Numéro d'assuré | |
|
||
| `SANG` | Groupe sanguin | 🆕 |
|
||
| `PERMIS` / `C` | Permis de séjour / établissem. | 🆕 |
|
||
| `PASSPORT` | Numéro de passeport | 🆕 |
|
||
| `SUI` / `ITA` | Suisse / Italie | 🆕 |
|
||
| `TITRE` / `MED` | Titre professionnel / médical | 🆕 |
|
||
| `DOSSIER` | Numéro de dossier | 🆕 |
|
||
|
||
**[→ See all 10 examples, with full `mapping` and `legend` tables](./examples/RESULTS.md)**
|
||
|
||
## How this helps with the nLPD
|
||
|
||
Switzerland's [nLPD](https://www.fedlex.admin.ch/eli/cc/2022/491/fr) (and the GDPR) push for **data
|
||
minimisation** and favour **pseudonymisation** when personal data is processed by third parties. This
|
||
library is built around those principles:
|
||
|
||
- **The third party never sees raw PII.** When you send text to an external LLM (or any external service),
|
||
it receives only pseudonymised tokens like `[PER_1.NOM:M]` plus the non-identifying `legend` — never the
|
||
real name, e-mail, AVS number, etc.
|
||
- **Pseudonymisation, not loss of meaning.** The re-identification key (`mapping`) stays in your
|
||
infrastructure; only you can reverse the tokens. The `legend` lets the downstream model still reason
|
||
correctly ("a person", "male") without knowing _who_.
|
||
- **Fail-closed option.** Omitting the regex fallback means that if detection can't run, the call errors
|
||
instead of forwarding data that wasn't pseudonymised — no silent leak.
|
||
- **Coreference & minimisation.** Re-using one id per entity avoids spreading extra distinguishing detail
|
||
across a prompt, and the corpus itself can stay in clear text — pseudonymisation happens only at the
|
||
boundary, at query time.
|
||
|
||
> This is an engineering aid, not legal advice or a certification. You remain the data controller; assess
|
||
> it against your own obligations (see the disclaimer below).
|
||
|
||
## Compliance note
|
||
|
||
This library is a **best-effort** pseudonymization aid, not a guarantee of legal compliance. LLM and regex
|
||
detection can miss or mis-classify data. Validate against your own requirements (nLPD, GDPR, HIPAA, …)
|
||
before relying on it for regulated data.
|
||
|
||
## License
|
||
|
||
[MIT](./LICENSE) © Mobiletic
|