171 lines
6.2 KiB
Markdown
171 lines
6.2 KiB
Markdown
# @mobiletic/anonymizer
|
||
|
||
[](https://git.mobiletic.net/mobiletic/anonymizer/actions)
|
||
[](https://www.npmjs.com/package/@mobiletic/anonymizer)
|
||
[](./LICENSE)
|
||
|
||
Framework-agnostic **PII anonymization & pseudonymization** for TypeScript/JavaScript.
|
||
|
||
It replaces personal data in text with stable placeholders before the text leaves your trust boundary
|
||
(e.g. before sending it to a third-party LLM, log sink, or analytics pipeline), and restores the real
|
||
values afterwards — including across **streamed** tokens.
|
||
|
||
- 🔌 **Pluggable LLM detection** — catch free-form PII (names, addresses) via any OpenAI-compatible
|
||
endpoint, or your own provider. No LLM? It degrades gracefully to regex.
|
||
- 🧩 **Deterministic regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
|
||
pattern presets (`swiss`, `generic`) or your own.
|
||
- 🔁 **Deterministic coreference** — the same person keeps the same id (`[PER_1]`) across a question and
|
||
every retrieved chunk, with automatic de-collision.
|
||
- 🌊 **Streaming-safe** — a placeholder split across two stream chunks (`[PER_` + `1.NOM:M]`) is never
|
||
leaked partially.
|
||
- 🪶 **Zero runtime dependencies**, ESM + CJS, fully typed.
|
||
|
||
> Built by [Mobiletic](https://mobiletic.com) and extracted from a production Swiss-nLPD chatbot.
|
||
|
||
## Install
|
||
|
||
```bash
|
||
npm install @mobiletic/anonymizer
|
||
```
|
||
|
||
Requires Node ≥ 18 (uses native `fetch`).
|
||
|
||
## Quick start
|
||
|
||
### Regex-only (no LLM, fully deterministic)
|
||
|
||
```ts
|
||
import { Anonymizer, presets } from '@mobiletic/anonymizer';
|
||
|
||
const anonymizer = new Anonymizer({ patterns: presets.swiss });
|
||
|
||
const { anon, mapping } = await anonymizer.anonymize('Écris à jean@exemple.ch');
|
||
// anon -> "Écris à [EMAIL_1]"
|
||
// mapping -> { "[EMAIL_1]": "jean@exemple.ch" }
|
||
|
||
anonymizer.deanonymize(anon, mapping); // -> "Écris à jean@exemple.ch"
|
||
```
|
||
|
||
### With an LLM (also catches names, addresses…)
|
||
|
||
```ts
|
||
import { Anonymizer, openAICompatibleProvider, presets } from '@mobiletic/anonymizer';
|
||
|
||
const anonymizer = new Anonymizer({
|
||
llm: openAICompatibleProvider({
|
||
baseUrl: process.env.LLM_BASE_URL!, // OpenAI, Infomaniak, vLLM, Ollama, …
|
||
apiKey: process.env.LLM_API_KEY!,
|
||
model: process.env.LLM_MODEL!,
|
||
timeoutMs: 3000,
|
||
}),
|
||
patterns: presets.swiss, // regex fallback if the LLM is down/misbehaves
|
||
});
|
||
|
||
const { anon, mapping } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
|
||
// anon -> "Le dossier de [PER_1.NOM:M] est complet."
|
||
```
|
||
|
||
If the LLM call fails, times out, or returns an invalid shape, the anonymizer automatically falls back to
|
||
the regex engine — it never throws on a provider failure.
|
||
|
||
`openAICompatibleProvider` retries **transient** failures (network error, timeout, HTTP 429/5xx) before
|
||
giving up; 4xx and malformed responses are not retried. Tune with `timeoutMs` (per attempt, default 3000),
|
||
`retries` (default 1 → 2 attempts), and `retryDelayMs` (linear backoff, default 250). Worst-case latency
|
||
is `(retries + 1) × timeoutMs`, so keep `retries` low on latency-sensitive paths.
|
||
|
||
### Streaming de-anonymization
|
||
|
||
When you stream an LLM answer back to a user, restore real values without ever emitting a half-written
|
||
placeholder:
|
||
|
||
```ts
|
||
const stream = anonymizer.makeStreamDeanonymizer(mapping);
|
||
for await (const token of llmTokens) process.stdout.write(stream.push(token));
|
||
process.stdout.write(stream.flush());
|
||
```
|
||
|
||
### Anonymizing retrieved chunks consistently (RAG)
|
||
|
||
`anonymizeChunks(chunks, seed)` reuses the question's mapping (`seed`) so the same person gets the same id
|
||
across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:
|
||
|
||
```ts
|
||
const q = await anonymizer.anonymize(question);
|
||
const { anon, mapping } = await anonymizer.anonymizeChunks(retrievedChunks, q.mapping);
|
||
// `mapping` is the full question ∪ chunks table; pass it to deanonymize()/makeStreamDeanonymizer().
|
||
```
|
||
|
||
## Configuration
|
||
|
||
```ts
|
||
new Anonymizer({
|
||
llm?, // LlmProvider — omit for regex-only mode
|
||
patterns?, // PatternDef[] — defaults to presets.swiss
|
||
nameHint?, // RegExp flagging likely names so the LLM is consulted (has a default)
|
||
logger?, // { warn(msg) } — receives fallback warnings; defaults to no-op
|
||
});
|
||
```
|
||
|
||
### Presets & custom patterns
|
||
|
||
```ts
|
||
import { presets } from '@mobiletic/anonymizer';
|
||
|
||
presets.swiss; // AVS, IBAN CH, EMAIL, Swiss phone, DATE
|
||
presets.generic; // EMAIL, IBAN, credit card, IPv4, phone, DATE
|
||
|
||
// Compose / extend:
|
||
const patterns = [
|
||
...presets.generic,
|
||
{ tag: 'TICKET', re: /\bJIRA-\d+\b/g }, // patterns must use the global flag
|
||
];
|
||
```
|
||
|
||
Each pattern may carry an optional `validate(match) => boolean` second stage — a match is only redacted
|
||
if it passes. `presets.generic` uses it for a [Luhn](https://en.wikipedia.org/wiki/Luhn_algorithm) check
|
||
so arbitrary long digit runs aren't mistaken for credit cards:
|
||
|
||
```ts
|
||
{ tag: 'CREDIT_CARD', re: /\b\d(?:[ -]?\d){12,18}\b/g, validate: luhnValid }
|
||
```
|
||
|
||
> The `generic` preset is a **best-effort** starting point — broad patterns (phone, date, card) can
|
||
> overlap. For production use, prefer a locale-specific preset (`presets.swiss`) or your own patterns.
|
||
|
||
### Custom LLM provider
|
||
|
||
Implement `LlmProvider` to use any backend (Anthropic, a local model, a rules engine…):
|
||
|
||
```ts
|
||
import type { LlmProvider } from '@mobiletic/anonymizer';
|
||
|
||
const myProvider: LlmProvider = {
|
||
isConfigured: () => true,
|
||
async anonymize(text) {
|
||
/* return { anon, mapping } */
|
||
},
|
||
async anonymizeBatch(texts, usedIds) {
|
||
/* return { segments, mapping } */
|
||
},
|
||
};
|
||
```
|
||
|
||
## Placeholder format
|
||
|
||
```
|
||
[PER_1.NOM:M] entity PER #1, attribute NOM, context M (rich, from the LLM)
|
||
[EMAIL_1] structured id from the regex fallback
|
||
```
|
||
|
||
`PLACEHOLDER_RE` is exported if you need to scan text for placeholders.
|
||
|
||
## Compliance note
|
||
|
||
This library is a **best-effort** pseudonymization aid, not a guarantee of legal compliance. LLM and regex
|
||
detection can miss or mis-classify data. Validate against your own requirements (nLPD, GDPR, HIPAA, …)
|
||
before relying on it for regulated data.
|
||
|
||
## License
|
||
|
||
[MIT](./LICENSE) © Mobiletic
|