feat!: optional fail-closed fallback, legend output, richer prompt & docs
BREAKING CHANGE: regex fallback is now opt-in (patterns no longer defaults
to presets.swiss). With no fallback an LLM failure throws AnonymizationError
(fail-closed) and the pre-filter is bypassed; at least one of llm/patterns is
required. AnonymizationResult gains a required `legend`; anonymizeChunks seed
is now { mapping, legend? } and returns legend.
- prompt: model may coin new UPPERCASE abbreviations and returns a 'legende'
explaining every abbreviation used (French); backfilled by DEFAULT_LEGEND
- PatternDef.meaning surfaces in the legend; swiss/generic presets get meanings
- AnonymizationError (exported) wraps the cause on fail-closed
- README: drop the chatbot provenance line; add 'How it works' + nLPD sections
- 34 tests / 99% coverage; bump to 0.2.0
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
25
CHANGELOG.md
25
CHANGELOG.md
@@ -4,11 +4,34 @@ All notable changes to this project are documented here. The format is based on
|
||||
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
|
||||
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||
|
||||
## [0.2.0] - Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- **`legend`** on every result — abbreviation → French meaning (`PER`→`Personne`, `M`→`Masculin`), safe to
|
||||
forward to a downstream LLM so it understands the placeholder tokens. Backed by a built-in
|
||||
`DEFAULT_LEGEND` so coverage is guaranteed even if the model omits entries.
|
||||
- The default prompt now lets the model **coin new uppercase abbreviations** for entities/attributes/
|
||||
context it discovers and return their meanings in `legende`.
|
||||
- `PatternDef.meaning` — optional human label for a tag, surfaced in the `legend`. `presets.swiss`/
|
||||
`presets.generic` ship French meanings.
|
||||
- `AnonymizationError` (exported) — thrown when anonymization can't complete and no fallback exists;
|
||||
carries the originating error in `.cause`.
|
||||
|
||||
### Changed (breaking)
|
||||
|
||||
- **Regex fallback is now opt-in.** `patterns` no longer defaults to `presets.swiss`. With no fallback,
|
||||
an LLM failure throws `AnonymizationError` (fail-closed) and the pre-filter is bypassed. At least one of
|
||||
`llm` or `patterns` is required, or the constructor throws.
|
||||
- `AnonymizationResult` gained a required `legend` field; `LlmProvider.anonymizeBatch` returns `legend`.
|
||||
- `anonymizeChunks(chunks, seed)` — `seed` is now `{ mapping, legend? }` (was the bare mapping) and the
|
||||
return includes `legend`.
|
||||
|
||||
## [0.1.0] - Unreleased
|
||||
|
||||
### Added
|
||||
|
||||
- Initial public release, extracted from Mobiletic's production Swiss-nLPD chatbot.
|
||||
- Initial public release.
|
||||
- `Anonymizer` — pre-filter → LLM → regex fallback, bidirectional validation, deterministic coreference,
|
||||
and de-collision across question and retrieved chunks (`anonymize`, `anonymizeChunks`, `deanonymize`).
|
||||
- `makeStreamDeanonymizer` — streaming-safe de-anonymization that never leaks a split placeholder.
|
||||
|
||||
107
README.md
107
README.md
@@ -11,16 +11,18 @@ It replaces personal data in text with stable placeholders before the text leave
|
||||
values afterwards — including across **streamed** tokens.
|
||||
|
||||
- 🔌 **Pluggable LLM detection** — catch free-form PII (names, addresses) via any OpenAI-compatible
|
||||
endpoint, or your own provider. No LLM? It degrades gracefully to regex.
|
||||
- 🧩 **Deterministic regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
|
||||
pattern presets (`swiss`, `generic`) or your own.
|
||||
endpoint, or your own provider.
|
||||
- 🧩 **Optional regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
|
||||
presets (`swiss`, `generic`) or your own. Opt in for graceful degradation, or omit it to **fail closed**.
|
||||
- 🏷️ **Self-describing tokens** — every result ships a `legend` (`PER`→`Personne`, `M`→`Masculin`) you can
|
||||
hand to the downstream LLM so it understands the placeholders; the model may coin new abbreviations too.
|
||||
- 🔁 **Deterministic coreference** — the same person keeps the same id (`[PER_1]`) across a question and
|
||||
every retrieved chunk, with automatic de-collision.
|
||||
- 🌊 **Streaming-safe** — a placeholder split across two stream chunks (`[PER_` + `1.NOM:M]`) is never
|
||||
leaked partially.
|
||||
- 🪶 **Zero runtime dependencies**, ESM + CJS, fully typed.
|
||||
|
||||
> Built by [Mobiletic](https://mobiletic.com) and extracted from a production Swiss-nLPD chatbot.
|
||||
> Built by [Mobiletic](https://mobiletic.com).
|
||||
|
||||
## Install
|
||||
|
||||
@@ -39,9 +41,10 @@ import { Anonymizer, presets } from '@mobiletic/anonymizer';
|
||||
|
||||
const anonymizer = new Anonymizer({ patterns: presets.swiss });
|
||||
|
||||
const { anon, mapping } = await anonymizer.anonymize('Écris à jean@exemple.ch');
|
||||
const { anon, mapping, legend } = await anonymizer.anonymize('Écris à jean@exemple.ch');
|
||||
// anon -> "Écris à [EMAIL_1]"
|
||||
// mapping -> { "[EMAIL_1]": "jean@exemple.ch" }
|
||||
// mapping -> { "[EMAIL_1]": "jean@exemple.ch" } (secret — keep on your side)
|
||||
// legend -> { "EMAIL": "Adresse e-mail" } (safe to share downstream)
|
||||
|
||||
anonymizer.deanonymize(anon, mapping); // -> "Écris à jean@exemple.ch"
|
||||
```
|
||||
@@ -58,15 +61,19 @@ const anonymizer = new Anonymizer({
|
||||
model: process.env.LLM_MODEL!,
|
||||
timeoutMs: 3000,
|
||||
}),
|
||||
patterns: presets.swiss, // regex fallback if the LLM is down/misbehaves
|
||||
patterns: presets.swiss, // OPTIONAL regex fallback if the LLM is down/misbehaves
|
||||
});
|
||||
|
||||
const { anon, mapping } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
|
||||
const { anon, mapping, legend } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
|
||||
// anon -> "Le dossier de [PER_1.NOM:M] est complet."
|
||||
// legend -> { "PER": "Personne", "NOM": "Nom de famille", "M": "Masculin" }
|
||||
```
|
||||
|
||||
If the LLM call fails, times out, or returns an invalid shape, the anonymizer automatically falls back to
|
||||
the regex engine — it never throws on a provider failure.
|
||||
**Fallback is opt-in (fail-closed).** If a `patterns` fallback is configured, a failed/timed-out/invalid
|
||||
LLM call degrades to the regex engine. If you omit `patterns`, there's nothing to degrade to, so the call
|
||||
**throws an `AnonymizationError`** (with the underlying error as `.cause`) — it never silently returns
|
||||
un-anonymized text. With no fallback the pre-filter is also bypassed, so every non-trivial call consults
|
||||
the LLM (more calls, no leaks). At least one of `llm` or `patterns` is required.
|
||||
|
||||
`openAICompatibleProvider` retries **transient** failures (network error, timeout, HTTP 429/5xx) before
|
||||
giving up; 4xx and malformed responses are not retried. Tune with `timeoutMs` (per attempt, default 3000),
|
||||
@@ -86,13 +93,14 @@ process.stdout.write(stream.flush());
|
||||
|
||||
### Anonymizing retrieved chunks consistently (RAG)
|
||||
|
||||
`anonymizeChunks(chunks, seed)` reuses the question's mapping (`seed`) so the same person gets the same id
|
||||
across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:
|
||||
`anonymizeChunks(chunks, seed)` reuses the question's `{ mapping, legend }` so the same person gets the
|
||||
same id across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:
|
||||
|
||||
```ts
|
||||
const q = await anonymizer.anonymize(question);
|
||||
const { anon, mapping } = await anonymizer.anonymizeChunks(retrievedChunks, q.mapping);
|
||||
// `mapping` is the full question ∪ chunks table; pass it to deanonymize()/makeStreamDeanonymizer().
|
||||
const { anon, mapping, legend } = await anonymizer.anonymizeChunks(retrievedChunks, q);
|
||||
// `mapping`/`legend` are the full question ∪ chunks tables;
|
||||
// pass `mapping` to deanonymize()/makeStreamDeanonymizer(), and `legend` to the downstream LLM.
|
||||
```
|
||||
|
||||
## Configuration
|
||||
@@ -100,10 +108,11 @@ const { anon, mapping } = await anonymizer.anonymizeChunks(retrievedChunks, q.ma
|
||||
```ts
|
||||
new Anonymizer({
|
||||
llm?, // LlmProvider — omit for regex-only mode
|
||||
patterns?, // PatternDef[] — defaults to presets.swiss
|
||||
patterns?, // PatternDef[] — opt-in regex fallback; omit to fail closed
|
||||
nameHint?, // RegExp flagging likely names so the LLM is consulted (has a default)
|
||||
logger?, // { warn(msg) } — receives fallback warnings; defaults to no-op
|
||||
});
|
||||
// At least one of `llm` or `patterns` must be provided, or the constructor throws.
|
||||
```
|
||||
|
||||
### Presets & custom patterns
|
||||
@@ -142,10 +151,10 @@ import type { LlmProvider } from '@mobiletic/anonymizer';
|
||||
const myProvider: LlmProvider = {
|
||||
isConfigured: () => true,
|
||||
async anonymize(text) {
|
||||
/* return { anon, mapping } */
|
||||
/* return { anon, mapping, legend } */
|
||||
},
|
||||
async anonymizeBatch(texts, usedIds) {
|
||||
/* return { segments, mapping } */
|
||||
/* return { segments, mapping, legend } */
|
||||
},
|
||||
};
|
||||
```
|
||||
@@ -157,7 +166,69 @@ const myProvider: LlmProvider = {
|
||||
[EMAIL_1] structured id from the regex fallback
|
||||
```
|
||||
|
||||
`PLACEHOLDER_RE` is exported if you need to scan text for placeholders.
|
||||
`PLACEHOLDER_RE` is exported if you need to scan text for placeholders. The LLM may also **coin new
|
||||
abbreviations** (always uppercase `[A-Z_]`) for entities/attributes/context it discovers — every one it
|
||||
uses is described in the result `legend`.
|
||||
|
||||
## How it works
|
||||
|
||||
The library does **query-time pseudonymization**: it rewrites text so that personal data never leaves your
|
||||
trust boundary in identifiable form, while keeping the answer fully reversible on your side.
|
||||
|
||||
```
|
||||
┌───────────────────────────── your trust boundary ──────────────────────────────┐
|
||||
raw text ─▶ ① pre-filter ─▶ ② LLM detect ─▶ ③ regex fallback ─▶ ④ validate ─▶ anon + mapping + legend
|
||||
(skip if (names, (optional; (every │ │
|
||||
clearly addresses, structured ids; placeholder ┌───────┘ │
|
||||
PII-free) coins abbrevs, fail closed if is mapped) ▼ ▼
|
||||
builds legend) absent) mapping legend
|
||||
(SECRET, (shareable:
|
||||
downstream LLM ◀── anon + legend ───────────────────────────────────────────── reversible) PER=Personne…)
|
||||
answer (with [PER_1.NOM:M] tokens)
|
||||
│
|
||||
▼
|
||||
⑤ stream de-anonymize ──▶ real values restored for the end user (placeholders never leak, even if split)
|
||||
```
|
||||
|
||||
1. **Pre-filter** — a cheap regex/name check skips the LLM round-trip for text that clearly has no PII.
|
||||
(Bypassed when no regex fallback is configured, so nothing slips through.)
|
||||
2. **LLM detection** — finds free-form PII a regex can't (names, addresses), keeps **coreference** (the
|
||||
same person is always `[PER_1]`), coins uppercase abbreviations for anything new, and returns a
|
||||
**legend** describing them.
|
||||
3. **Regex fallback** — optional, deterministic detection of structured identifiers; used if the LLM is
|
||||
unavailable. Omit it to **fail closed** (raise `AnonymizationError` rather than risk a leak).
|
||||
4. **Validation** — bidirectional check that every placeholder has a mapping entry and vice-versa.
|
||||
5. **Streaming de-anonymization** — `makeStreamDeanonymizer` restores real values token-by-token, buffering
|
||||
any placeholder split across chunks so a partial `[PER_` is never emitted.
|
||||
|
||||
Three distinct outputs, with different sensitivities:
|
||||
|
||||
| Output | Example | Sensitivity |
|
||||
| --------- | ---------------------------------------- | ------------------------------------------------------- |
|
||||
| `anon` | `Le dossier de [PER_1.NOM:M]` | Safe to send onward — no identifiers |
|
||||
| `mapping` | `{ "[PER_1.NOM:M]": "Alain Jaccard" }` | **Secret** — re-identifies people; keep it on your side |
|
||||
| `legend` | `{ "PER": "Personne", "M": "Masculin" }` | Safe to share — explains the tokens to a downstream LLM |
|
||||
|
||||
## How this helps with the nLPD
|
||||
|
||||
Switzerland's [nLPD](https://www.fedlex.admin.ch/eli/cc/2022/491/fr) (and the GDPR) push for **data
|
||||
minimisation** and favour **pseudonymisation** when personal data is processed by third parties. This
|
||||
library is built around those principles:
|
||||
|
||||
- **The third party never sees raw PII.** When you send text to an external LLM (or any external service),
|
||||
it receives only pseudonymised tokens like `[PER_1.NOM:M]` plus the non-identifying `legend` — never the
|
||||
real name, e-mail, AVS number, etc.
|
||||
- **Pseudonymisation, not loss of meaning.** The re-identification key (`mapping`) stays in your
|
||||
infrastructure; only you can reverse the tokens. The `legend` lets the downstream model still reason
|
||||
correctly ("a person", "male") without knowing _who_.
|
||||
- **Fail-closed option.** Omitting the regex fallback means that if detection can't run, the call errors
|
||||
instead of forwarding data that wasn't pseudonymised — no silent leak.
|
||||
- **Coreference & minimisation.** Re-using one id per entity avoids spreading extra distinguishing detail
|
||||
across a prompt, and the corpus itself can stay in clear text — pseudonymisation happens only at the
|
||||
boundary, at query time.
|
||||
|
||||
> This is an engineering aid, not legal advice or a certification. You remain the data controller; assess
|
||||
> it against your own obligations (see the disclaimer below).
|
||||
|
||||
## Compliance note
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "@mobiletic/anonymizer",
|
||||
"version": "0.1.0",
|
||||
"version": "0.2.0",
|
||||
"description": "Framework-agnostic PII anonymization & pseudonymization: pluggable LLM detection for free-form PII (names, addresses) with a deterministic regex fallback, deterministic coreference, and streaming-safe de-anonymization.",
|
||||
"license": "MIT",
|
||||
"author": "Mobiletic",
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { RegexFallback } from './fallback.js';
|
||||
import { presets } from './presets.js';
|
||||
import { AnonymizationError } from './errors.js';
|
||||
import {
|
||||
type AnonymizationResult,
|
||||
type AnonymizerConfig,
|
||||
@@ -14,6 +14,36 @@ const NOOP_LOGGER: Logger = { warn: () => {} };
|
||||
// Likely proper-name heuristic (two capitalized words), in addition to the structured fallback.
|
||||
const DEFAULT_NAME_HINT = /\b[A-ZÀ-Ý][a-zà-ÿ]+\s+[A-ZÀ-Ý][a-zà-ÿ]+\b/;
|
||||
|
||||
/**
|
||||
* Built-in French meanings for the standard abbreviations, used to backfill the
|
||||
* legend when the model (or the regex fallback) doesn't supply one for an
|
||||
* abbreviation that actually appears in the output.
|
||||
*/
|
||||
const DEFAULT_LEGEND: Record<string, string> = {
|
||||
PER: 'Personne',
|
||||
ORG: 'Organisation',
|
||||
LOC: 'Lieu',
|
||||
NOM: 'Nom de famille',
|
||||
PRENOM: 'Prénom',
|
||||
DATE_NAISSANCE: 'Date de naissance',
|
||||
AGE: 'Âge',
|
||||
ADRESSE: 'Adresse',
|
||||
EMAIL: 'Adresse e-mail',
|
||||
TELEPHONE: 'Numéro de téléphone',
|
||||
TEL: 'Numéro de téléphone',
|
||||
AVS: 'Numéro AVS',
|
||||
IBAN: 'IBAN',
|
||||
NSS: 'Numéro de sécurité sociale',
|
||||
DATE: 'Date',
|
||||
IPV4: 'Adresse IP',
|
||||
CREDIT_CARD: 'Carte de crédit',
|
||||
M: 'Masculin',
|
||||
F: 'Féminin',
|
||||
U: 'Inconnu',
|
||||
Mineur: 'Mineur',
|
||||
Adulte: 'Adulte',
|
||||
};
|
||||
|
||||
/**
|
||||
* Return a stateless copy of a regex: a global/sticky (`g`/`y`) regex carries a
|
||||
* mutable `lastIndex`, which makes repeated `.test()` calls return alternating
|
||||
@@ -27,24 +57,31 @@ function stateless(re: RegExp): RegExp {
|
||||
* Framework-agnostic anonymization engine.
|
||||
*
|
||||
* `anonymize()`: pre-filter (skips the LLM when there is no PII) → LLM →
|
||||
* regex fallback on failure/timeout → bidirectional validation.
|
||||
* regex fallback on failure/timeout → bidirectional validation. When no fallback
|
||||
* is configured, an LLM failure throws {@link AnonymizationError} (fail-closed).
|
||||
* `deanonymize()` restores the real values. `makeStreamDeanonymizer()` handles
|
||||
* the critical case of a placeholder fragmented across several streamed tokens.
|
||||
*
|
||||
* Pass an {@link LlmProvider} to detect free-form PII (names, addresses);
|
||||
* omit it for deterministic regex-only mode.
|
||||
* Each result carries a `legend` (abbreviation → meaning) you can forward to a
|
||||
* downstream LLM so it understands the placeholder tokens.
|
||||
*
|
||||
* Pass an {@link LlmProvider} to detect free-form PII (names, addresses) and/or
|
||||
* `patterns` for the regex fallback. At least one of the two is required.
|
||||
*/
|
||||
export class Anonymizer {
|
||||
private readonly llm?: LlmProvider;
|
||||
private readonly fallback: RegexFallback;
|
||||
private readonly fallback?: RegexFallback;
|
||||
private readonly nameHint: RegExp;
|
||||
private readonly logger: Logger;
|
||||
|
||||
constructor(config: AnonymizerConfig = {}) {
|
||||
this.llm = config.llm;
|
||||
this.fallback = new RegexFallback(config.patterns ?? presets.swiss);
|
||||
this.fallback = config.patterns ? new RegexFallback(config.patterns) : undefined;
|
||||
this.nameHint = stateless(config.nameHint ?? DEFAULT_NAME_HINT);
|
||||
this.logger = config.logger ?? NOOP_LOGGER;
|
||||
if (!this.llm && !this.fallback) {
|
||||
throw new AnonymizationError('Anonymizer requires an LlmProvider, regex patterns, or both');
|
||||
}
|
||||
}
|
||||
|
||||
private hasLlm(): boolean {
|
||||
@@ -52,51 +89,67 @@ export class Anonymizer {
|
||||
}
|
||||
|
||||
private likelyHasPii(text: string): boolean {
|
||||
return this.fallback.hasPii(text) || this.nameHint.test(text);
|
||||
return (this.fallback?.hasPii(text) ?? false) || this.nameHint.test(text);
|
||||
}
|
||||
|
||||
async anonymize(text: string): Promise<AnonymizationResult> {
|
||||
// Pre-filter: most text has no PII → avoid the LLM round-trip (latency + cost)
|
||||
// and return the text unchanged.
|
||||
if (!this.likelyHasPii(text)) return { anon: text, mapping: {} };
|
||||
// Pre-filter: most text has no PII → avoid the LLM round-trip. Only safe when
|
||||
// a regex fallback exists to judge "no PII"; without one we always consult the
|
||||
// LLM so structured PII (e.g. emails) can't slip through unanonymized.
|
||||
if (this.fallback && !this.likelyHasPii(text)) return { anon: text, mapping: {}, legend: {} };
|
||||
|
||||
if (this.hasLlm()) {
|
||||
try {
|
||||
const result = await this.llm!.anonymize(text);
|
||||
this.validate(result); // throws if inconsistent → fall back
|
||||
return result;
|
||||
return {
|
||||
anon: result.anon,
|
||||
mapping: result.mapping,
|
||||
legend: this.buildLegend(result.anon, result.legend),
|
||||
};
|
||||
} catch (err) {
|
||||
if (!this.fallback) {
|
||||
throw new AnonymizationError(`anonymize failed: ${(err as Error).message}`, { cause: err });
|
||||
}
|
||||
this.logger.warn(`LLM unavailable/invalid → regex fallback: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
return this.fallback.anonymize(text);
|
||||
return this.fallbackAnonymize(text);
|
||||
}
|
||||
|
||||
/**
|
||||
* Anonymize retrieved chunks BEFORE sending them on, CONSISTENTLY with the
|
||||
* question's mapping (`seed`). Returns the anonymized chunks + the COMPLETE
|
||||
* mapping (question ∪ chunks). Strategy:
|
||||
* question's `seed` ({ mapping, legend }). Returns the anonymized chunks + the
|
||||
* COMPLETE mapping and legend (question ∪ chunks). Strategy:
|
||||
* 1. Deterministic reuse: replace any already-known value (from the question)
|
||||
* by its placeholder → question↔chunk coreference with no LLM call.
|
||||
* 2. Pre-filter: if no residual PII remains → return as-is (most corpus chunks
|
||||
* have no PII → LLM cost avoided).
|
||||
* have no PII → LLM cost avoided). Skipped when no regex fallback is set.
|
||||
* 3. Otherwise: ONE batched LLM call, then de-collision merge (renumber the
|
||||
* genuinely new entities). Failure/timeout → regex fallback per chunk.
|
||||
* genuinely new entities). Failure/timeout → regex fallback per chunk, or
|
||||
* {@link AnonymizationError} when no fallback is configured.
|
||||
*/
|
||||
async anonymizeChunks(
|
||||
chunks: string[],
|
||||
seed: Record<string, string>,
|
||||
): Promise<{ anon: string[]; mapping: Record<string, string> }> {
|
||||
if (!chunks.length) return { anon: [], mapping: { ...seed } };
|
||||
seed: { mapping: Record<string, string>; legend?: Record<string, string> },
|
||||
): Promise<{ anon: string[]; mapping: Record<string, string>; legend: Record<string, string> }> {
|
||||
const seedLegend = seed.legend ?? {};
|
||||
if (!chunks.length) return { anon: [], mapping: { ...seed.mapping }, legend: { ...seedLegend } };
|
||||
|
||||
const seeded = chunks.map((c) => this.applyKnown(c, seed));
|
||||
if (!seeded.some((c) => this.likelyHasPii(c))) return { anon: seeded, mapping: { ...seed } };
|
||||
const seeded = chunks.map((c) => this.applyKnown(c, seed.mapping));
|
||||
if (this.fallback && !seeded.some((c) => this.likelyHasPii(c))) {
|
||||
return { anon: seeded, mapping: { ...seed.mapping }, legend: { ...seedLegend } };
|
||||
}
|
||||
|
||||
if (this.hasLlm()) {
|
||||
try {
|
||||
const { segments, mapping: gMap } = await this.llm!.anonymizeBatch(seeded, Object.keys(seed));
|
||||
const {
|
||||
segments,
|
||||
mapping: gMap,
|
||||
legend: gLegend,
|
||||
} = await this.llm!.anonymizeBatch(seeded, Object.keys(seed.mapping));
|
||||
if (segments.length !== seeded.length) throw new Error('SEGMENT_COUNT_MISMATCH');
|
||||
const { mapping, rename } = this.mergeMappings(seed, gMap);
|
||||
const { mapping, rename } = this.mergeMappings(seed.mapping, gMap);
|
||||
const anon = segments.map((s) => this.applyRename(s, rename));
|
||||
// Anti-leak: every placeholder present in a chunk must be mapped.
|
||||
for (const s of anon) {
|
||||
@@ -104,12 +157,16 @@ export class Anonymizer {
|
||||
if (!(ph in mapping)) throw new Error(`unmapped placeholder ${ph}`);
|
||||
}
|
||||
}
|
||||
return { anon, mapping };
|
||||
const legend = this.buildLegend(anon.join('\n'), { ...seedLegend, ...gLegend });
|
||||
return { anon, mapping, legend };
|
||||
} catch (err) {
|
||||
if (!this.fallback) {
|
||||
throw new AnonymizationError(`anonymizeChunks failed: ${(err as Error).message}`, { cause: err });
|
||||
}
|
||||
this.logger.warn(`Chunk anonymization → regex fallback: ${(err as Error).message}`);
|
||||
}
|
||||
}
|
||||
return this.fallbackChunks(seeded, seed);
|
||||
return this.fallbackChunks(seeded, seed.mapping, seedLegend);
|
||||
}
|
||||
|
||||
/** Replace known values (seed) by their placeholder, longest values first. */
|
||||
@@ -135,6 +192,32 @@ export class Anonymizer {
|
||||
return m ? { type: m[1], index: Number(m[2]), suffix: m[3] ?? '' } : null;
|
||||
}
|
||||
|
||||
/** Abbreviations used by a placeholder: entity type, attribute, and all-caps context codes. */
|
||||
private usedAbbreviations(text: string): Set<string> {
|
||||
const out = new Set<string>();
|
||||
for (const ph of text.match(PLACEHOLDER_RE) ?? []) {
|
||||
const m = /^\[([A-Z]+)_\d+(?:\.([A-Z_]+):([^\]]+))?\]$/.exec(ph);
|
||||
if (!m) continue;
|
||||
out.add(m[1]); // entity type
|
||||
if (m[2]) out.add(m[2]); // attribute
|
||||
if (m[3] && /^[A-Z]+$/.test(m[3])) out.add(m[3]); // context, only if it's a code (not a value)
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Build a legend covering every abbreviation that appears in `text`, taking the
|
||||
* meaning from `provided` (LLM/fallback) → {@link DEFAULT_LEGEND} → the
|
||||
* abbreviation itself. Guarantees coverage even if the model omits entries.
|
||||
*/
|
||||
private buildLegend(text: string, provided: Record<string, string>): Record<string, string> {
|
||||
const legend: Record<string, string> = {};
|
||||
for (const ab of this.usedAbbreviations(text)) {
|
||||
legend[ab] = provided[ab] ?? DEFAULT_LEGEND[ab] ?? ab;
|
||||
}
|
||||
return legend;
|
||||
}
|
||||
|
||||
/**
|
||||
* Merge the chunks' LLM mapping into the question's (seed).
|
||||
* - value already known → reuse the question's placeholder (rename).
|
||||
@@ -181,20 +264,29 @@ export class Anonymizer {
|
||||
return { mapping, rename };
|
||||
}
|
||||
|
||||
/** Single-text regex fallback, routed through the legend builder. */
|
||||
private fallbackAnonymize(text: string): AnonymizationResult {
|
||||
const r = this.fallback!.anonymize(text);
|
||||
return { anon: r.anon, mapping: r.mapping, legend: this.buildLegend(r.anon, r.legend) };
|
||||
}
|
||||
|
||||
/** Fallback (no LLM): regex per chunk, de-collision merged into the seed. */
|
||||
private fallbackChunks(
|
||||
seeded: string[],
|
||||
seed: Record<string, string>,
|
||||
): { anon: string[]; mapping: Record<string, string> } {
|
||||
seedLegend: Record<string, string>,
|
||||
): { anon: string[]; mapping: Record<string, string>; legend: Record<string, string> } {
|
||||
let mapping: Record<string, string> = { ...seed };
|
||||
let provided: Record<string, string> = { ...seedLegend };
|
||||
const anon: string[] = [];
|
||||
for (const c of seeded) {
|
||||
const r = this.fallback.anonymize(c);
|
||||
const r = this.fallback!.anonymize(c);
|
||||
const { mapping: merged, rename } = this.mergeMappings(mapping, r.mapping);
|
||||
mapping = merged;
|
||||
provided = { ...provided, ...r.legend };
|
||||
anon.push(this.applyRename(r.anon, rename));
|
||||
}
|
||||
return { anon, mapping };
|
||||
return { anon, mapping, legend: this.buildLegend(anon.join('\n'), provided) };
|
||||
}
|
||||
|
||||
/** Every placeholder in the text must be in the mapping AND vice-versa (anti-leak). */
|
||||
|
||||
11
src/errors.ts
Normal file
11
src/errors.ts
Normal file
@@ -0,0 +1,11 @@
|
||||
/**
|
||||
* Thrown when anonymization cannot be completed and there is no fallback to
|
||||
* degrade to — a fail-closed signal so the caller never proceeds with
|
||||
* un-anonymized text. The originating error is preserved in `.cause`.
|
||||
*/
|
||||
export class AnonymizationError extends Error {
|
||||
constructor(message: string, options?: { cause?: unknown }) {
|
||||
super(message, options);
|
||||
this.name = 'AnonymizationError';
|
||||
}
|
||||
}
|
||||
@@ -13,10 +13,11 @@ export class RegexFallback {
|
||||
|
||||
anonymize(text: string): AnonymizationResult {
|
||||
const mapping: Record<string, string> = {};
|
||||
const legend: Record<string, string> = {};
|
||||
const counters: Record<string, number> = {};
|
||||
let anon = text;
|
||||
|
||||
for (const { tag, re, validate } of this.patterns) {
|
||||
for (const { tag, re, validate, meaning } of this.patterns) {
|
||||
anon = anon.replace(re, (match) => {
|
||||
// A validator can reject a regex match (e.g. Luhn) → leave the text as-is.
|
||||
if (validate && !validate(match)) return match;
|
||||
@@ -26,10 +27,11 @@ export class RegexFallback {
|
||||
counters[tag] = (counters[tag] ?? 0) + 1;
|
||||
const ph = `[${tag}_${counters[tag]}]`;
|
||||
mapping[ph] = match;
|
||||
legend[tag] = meaning ?? tag; // document the abbreviation used
|
||||
return ph;
|
||||
});
|
||||
}
|
||||
return { anon, mapping };
|
||||
return { anon, mapping, legend };
|
||||
}
|
||||
|
||||
/** Fast pre-check: is there any structured identifier worth anonymizing? */
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
export { Anonymizer } from './anonymizer.js';
|
||||
export { AnonymizationError } from './errors.js';
|
||||
export { RegexFallback } from './fallback.js';
|
||||
export { presets } from './presets.js';
|
||||
export {
|
||||
|
||||
@@ -5,11 +5,11 @@ import type { PatternDef } from './types.js';
|
||||
* Order matters: specific patterns (AVS, IBAN) come before generic ones.
|
||||
*/
|
||||
const swiss: PatternDef[] = [
|
||||
{ tag: 'AVS', re: /\b756\.\d{4}\.\d{4}\.\d{2}\b/g }, // Swiss social security (AVS/AHV)
|
||||
{ tag: 'IBAN', re: /\bCH\d{2}[0-9A-Z]{17}\b/gi }, // Swiss IBAN
|
||||
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g },
|
||||
{ tag: 'TEL', re: /(?:\+41|0)(?:[\s.-]?\d){9}\b/g }, // Swiss phone number
|
||||
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
||||
{ tag: 'AVS', re: /\b756\.\d{4}\.\d{4}\.\d{2}\b/g, meaning: 'Numéro AVS' }, // Swiss social security
|
||||
{ tag: 'IBAN', re: /\bCH\d{2}[0-9A-Z]{17}\b/gi, meaning: 'IBAN' }, // Swiss IBAN
|
||||
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g, meaning: 'Adresse e-mail' },
|
||||
{ tag: 'TEL', re: /(?:\+41|0)(?:[\s.-]?\d){9}\b/g, meaning: 'Numéro de téléphone' }, // Swiss phone
|
||||
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g, meaning: 'Date' },
|
||||
];
|
||||
|
||||
/** Luhn check on a credit-card candidate (ignores spaces/dashes); 13–19 digits. */
|
||||
@@ -39,16 +39,20 @@ function luhnValid(value: string): boolean {
|
||||
* claimed before phone numbers so they don't get mis-tagged as `TEL`.
|
||||
*/
|
||||
const generic: PatternDef[] = [
|
||||
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g },
|
||||
{ tag: 'IBAN', re: /\b[A-Z]{2}\d{2}[0-9A-Z]{11,30}\b/g },
|
||||
{ tag: 'IPV4', re: /\b(?:\d{1,3}\.){3}\d{1,3}\b/g },
|
||||
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g, meaning: 'Adresse e-mail' },
|
||||
{ tag: 'IBAN', re: /\b[A-Z]{2}\d{2}[0-9A-Z]{11,30}\b/g, meaning: 'IBAN' },
|
||||
{ tag: 'IPV4', re: /\b(?:\d{1,3}\.){3}\d{1,3}\b/g, meaning: 'Adresse IP' },
|
||||
// 13–19 digits in optional space/dash groups, gated by Luhn to avoid eating
|
||||
// arbitrary long digit runs (account numbers, ids).
|
||||
{ tag: 'CREDIT_CARD', re: /\b\d(?:[ -]?\d){12,18}\b/g, validate: luhnValid },
|
||||
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
||||
{ tag: 'CREDIT_CARD', re: /\b\d(?:[ -]?\d){12,18}\b/g, validate: luhnValid, meaning: 'Carte de crédit' },
|
||||
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g, meaning: 'Date' },
|
||||
// Requires grouped/separated digits (or a leading +country) so plain integers
|
||||
// aren't mistaken for phone numbers.
|
||||
{ tag: 'TEL', re: /(?:\+\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]){2,3}\d{2,4}/g },
|
||||
{
|
||||
tag: 'TEL',
|
||||
re: /(?:\+\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]){2,3}\d{2,4}/g,
|
||||
meaning: 'Numéro de téléphone',
|
||||
},
|
||||
];
|
||||
|
||||
/** Built-in pattern sets for the regex fallback. */
|
||||
|
||||
@@ -13,12 +13,24 @@ export const DEFAULT_SYSTEM_PROMPT = [
|
||||
' NOM:M|F|U · DATE_NAISSANCE:<année> · AGE:Mineur|Adulte',
|
||||
' ADRESSE:Lieu|Rue|Ville|NPA|Pays · ORG:Entreprise|Ecole',
|
||||
'',
|
||||
'ABRÉVIATIONS DYNAMIQUES:',
|
||||
'- Si tu rencontres un TYPE d’entité, un ATTRIBUT ou un CONTEXTE non listé ci-dessus,',
|
||||
' INVENTE une abréviation COURTE en MAJUSCULES (lettres A–Z et "_" uniquement, ex. PER, NOM, M),',
|
||||
' suivant la même convention, et réutilise-la de façon cohérente partout.',
|
||||
'- N’utilise JAMAIS d’espaces, accents, chiffres ou symboles dans une abréviation.',
|
||||
'',
|
||||
'LÉGENDE:',
|
||||
'- Renvoie une "legende" : un objet associant CHAQUE abréviation utilisée (entité, attribut, contexte)',
|
||||
' à sa signification complète en français. Ex.: {"PER":"Personne","NOM":"Nom de famille","M":"Masculin"}.',
|
||||
'- La légende sert à expliquer les placeholders à un autre modèle ; elle ne contient AUCUNE donnée réelle.',
|
||||
'',
|
||||
'RÈGLES:',
|
||||
'1. Coréférence: la MÊME personne garde le MÊME identifiant (PER_1) dans tout le texte.',
|
||||
'2. N’anonymise JAMAIS les termes pédagogiques/techniques (langages, concepts, titres de cours, fonctions).',
|
||||
'3. Si AUCUNE donnée personnelle: renvoie le texte original et "mapping": {}.',
|
||||
'3. Si AUCUNE donnée personnelle: renvoie le texte original, "mapping": {} et "legende": {}.',
|
||||
'4. N’anonymise pas un placeholder déjà présent (idempotence).',
|
||||
'5. Sortie STRICTEMENT JSON valide: {"texte_anonymise": "...", "mapping": {"[PER_1.NOM:M]": "..."}}.',
|
||||
'5. Sortie STRICTEMENT JSON valide:',
|
||||
' {"texte_anonymise": "...", "mapping": {"[PER_1.NOM:M]": "..."}, "legende": {"PER": "Personne", "NOM": "Nom de famille", "M": "Masculin"}}.',
|
||||
].join('\n');
|
||||
|
||||
export interface OpenAICompatibleOptions {
|
||||
@@ -48,6 +60,20 @@ const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout
|
||||
/** A transport error that the caller may retry (network / timeout / 429 / 5xx). */
|
||||
class TransientError extends Error {}
|
||||
|
||||
/**
|
||||
* Coerce the model's `legende` into a string→string map, lenient by design: a
|
||||
* missing or malformed legend yields `{}` rather than failing the anonymization
|
||||
* (the legend is metadata; the `mapping` is what guarantees no leak).
|
||||
*/
|
||||
function asLegend(raw: unknown): Record<string, string> {
|
||||
if (!raw || typeof raw !== 'object') return {};
|
||||
const out: Record<string, string> = {};
|
||||
for (const [k, v] of Object.entries(raw as Record<string, unknown>)) {
|
||||
if (typeof v === 'string') out[k] = v;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
||||
'\n\nMODE LOT (segments) :' +
|
||||
'\n- ENTRÉE : un objet JSON {"segments": ["…", "…"]}.' +
|
||||
@@ -55,8 +81,8 @@ const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
||||
(usedIds.length
|
||||
? `\n- Identifiants DÉJÀ attribués (réutilise-les pour les mêmes valeurs, n'en duplique AUCUN pour d'autres valeurs) : ${usedIds.join(', ')}.`
|
||||
: '') +
|
||||
'\n- SORTIE STRICTEMENT JSON : {"segments": ["…anonymisé…"], "mapping": {"[PER_1.NOM:M]": "…"}}.' +
|
||||
'\n- "segments" DOIT avoir la même longueur et le même ordre que l’entrée ; "mapping" ne contient que les NOUVELLES entités.';
|
||||
'\n- SORTIE STRICTEMENT JSON : {"segments": ["…anonymisé…"], "mapping": {"[PER_1.NOM:M]": "…"}, "legende": {"PER": "Personne"}}.' +
|
||||
'\n- "segments" DOIT avoir la même longueur et le même ordre que l’entrée ; "mapping" ne contient que les NOUVELLES entités ; "legende" décrit toutes les abréviations utilisées.';
|
||||
|
||||
/**
|
||||
* Build an {@link LlmProvider} backed by any OpenAI-compatible Chat Completions
|
||||
@@ -131,7 +157,11 @@ export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProv
|
||||
|
||||
async anonymize(text: string): Promise<AnonymizationResult> {
|
||||
const content = await chat(systemPrompt, text);
|
||||
const parsed = JSON.parse(content) as { texte_anonymise?: string; mapping?: Record<string, string> };
|
||||
const parsed = JSON.parse(content) as {
|
||||
texte_anonymise?: string;
|
||||
mapping?: Record<string, string>;
|
||||
legende?: Record<string, string>;
|
||||
};
|
||||
if (
|
||||
typeof parsed.texte_anonymise !== 'string' ||
|
||||
typeof parsed.mapping !== 'object' ||
|
||||
@@ -139,20 +169,24 @@ export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProv
|
||||
) {
|
||||
throw new Error('LLM_BAD_SHAPE');
|
||||
}
|
||||
return { anon: parsed.texte_anonymise, mapping: parsed.mapping };
|
||||
return { anon: parsed.texte_anonymise, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||||
},
|
||||
|
||||
async anonymizeBatch(
|
||||
texts: string[],
|
||||
usedIds: string[],
|
||||
): Promise<{ segments: string[]; mapping: Record<string, string> }> {
|
||||
): Promise<{ segments: string[]; mapping: Record<string, string>; legend: Record<string, string> }> {
|
||||
const system = systemPrompt + batchInstructions(usedIds);
|
||||
const content = await chat(system, JSON.stringify({ segments: texts }));
|
||||
const parsed = JSON.parse(content) as { segments?: string[]; mapping?: Record<string, string> };
|
||||
const parsed = JSON.parse(content) as {
|
||||
segments?: string[];
|
||||
mapping?: Record<string, string>;
|
||||
legende?: Record<string, string>;
|
||||
};
|
||||
if (!Array.isArray(parsed.segments) || typeof parsed.mapping !== 'object' || !parsed.mapping) {
|
||||
throw new Error('LLM_BATCH_BAD_SHAPE');
|
||||
}
|
||||
return { segments: parsed.segments, mapping: parsed.mapping };
|
||||
return { segments: parsed.segments, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
26
src/types.ts
26
src/types.ts
@@ -1,9 +1,17 @@
|
||||
/** Result of an anonymization: text with placeholders + the reverse-mapping table. */
|
||||
/** Result of an anonymization: text with placeholders + the reverse-mapping table + a legend. */
|
||||
export interface AnonymizationResult {
|
||||
/** The text with every detected PII value replaced by a placeholder. */
|
||||
anon: string;
|
||||
/** Placeholder → original value, e.g. `{ "[PER_1.NOM:M]": "Alain JACCARD" }`. */
|
||||
/**
|
||||
* Placeholder → original value, e.g. `{ "[PER_1.NOM:M]": "Alain JACCARD" }`.
|
||||
* SECRET: it re-identifies people. Keep it on your side; never share it downstream.
|
||||
*/
|
||||
mapping: Record<string, string>;
|
||||
/**
|
||||
* Abbreviation → human meaning, e.g. `{ "PER": "Personne", "NOM": "Nom", "M": "Masculin" }`.
|
||||
* NON-secret: safe to send to a downstream LLM so it understands the placeholder tokens.
|
||||
*/
|
||||
legend: Record<string, string>;
|
||||
}
|
||||
|
||||
/**
|
||||
@@ -31,7 +39,7 @@ export interface LlmProvider {
|
||||
anonymizeBatch(
|
||||
texts: string[],
|
||||
usedIds: string[],
|
||||
): Promise<{ segments: string[]; mapping: Record<string, string> }>;
|
||||
): Promise<{ segments: string[]; mapping: Record<string, string>; legend: Record<string, string> }>;
|
||||
}
|
||||
|
||||
/** A structured-PII detector: a tag (e.g. `EMAIL`) and the global regex that finds it. */
|
||||
@@ -45,6 +53,11 @@ export interface PatternDef {
|
||||
* Use it to cut false positives (e.g. a Luhn check on credit-card candidates).
|
||||
*/
|
||||
validate?: (match: string) => boolean;
|
||||
/**
|
||||
* Optional human meaning for this tag, surfaced in the result `legend`
|
||||
* (e.g. `'Adresse e-mail'` for tag `EMAIL`). Falls back to the tag itself.
|
||||
*/
|
||||
meaning?: string;
|
||||
}
|
||||
|
||||
/** Minimal logger sink. Defaults to a no-op; pass your own to capture fallback warnings. */
|
||||
@@ -56,7 +69,12 @@ export interface Logger {
|
||||
export interface AnonymizerConfig {
|
||||
/** LLM backend for free-form PII (names, addresses…). Omit for regex-only mode. */
|
||||
llm?: LlmProvider;
|
||||
/** Structured-PII patterns for the regex fallback. Defaults to {@link presets.swiss}. */
|
||||
/**
|
||||
* Structured-PII patterns for the regex fallback (e.g. {@link presets.swiss}).
|
||||
* Opt-in: omit to disable the fallback — then an LLM failure throws
|
||||
* {@link AnonymizationError} (fail-closed) instead of degrading. At least one of
|
||||
* `llm` or `patterns` must be provided.
|
||||
*/
|
||||
patterns?: PatternDef[];
|
||||
/**
|
||||
* Heuristic that flags likely proper names so the LLM is consulted. Has a
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
import { describe, it, expect, vi } from 'vitest';
|
||||
import { Anonymizer, presets, type LlmProvider } from '../src/index.js';
|
||||
import { Anonymizer, AnonymizationError, presets, type LlmProvider } from '../src/index.js';
|
||||
|
||||
/** An LLM provider that is configured but always fails → forces the regex fallback. */
|
||||
const failingLlm = (overrides: Partial<LlmProvider> = {}): LlmProvider => ({
|
||||
@@ -50,7 +50,7 @@ describe('Anonymizer (Swiss preset)', () => {
|
||||
it('PII-free chunks → no LLM call, returned as-is', async () => {
|
||||
const llm = failingLlm();
|
||||
const s = new Anonymizer({ llm, patterns: presets.swiss });
|
||||
const r = await s.anonymizeChunks(['Un INNER JOIN combine deux tables.'], {});
|
||||
const r = await s.anonymizeChunks(['Un INNER JOIN combine deux tables.'], { mapping: {} });
|
||||
expect(llm.anonymizeBatch).not.toHaveBeenCalled();
|
||||
expect(r.anon[0]).toContain('INNER JOIN');
|
||||
expect(r.mapping).toEqual({});
|
||||
@@ -60,7 +60,7 @@ describe('Anonymizer (Swiss preset)', () => {
|
||||
const llm = failingLlm();
|
||||
const s = new Anonymizer({ llm, patterns: presets.swiss });
|
||||
const seed = { '[PER_1.NOM:M]': 'Alain JACCARD' };
|
||||
const r = await s.anonymizeChunks(['Le dossier de Alain JACCARD est complet.'], seed);
|
||||
const r = await s.anonymizeChunks(['Le dossier de Alain JACCARD est complet.'], { mapping: seed });
|
||||
expect(llm.anonymizeBatch).not.toHaveBeenCalled();
|
||||
expect(r.anon[0]).toContain('[PER_1.NOM:M]');
|
||||
expect(r.anon[0]).not.toContain('Alain JACCARD');
|
||||
@@ -71,19 +71,22 @@ describe('Anonymizer (Swiss preset)', () => {
|
||||
const anonymizeBatch = vi.fn().mockResolvedValue({
|
||||
segments: ['[PER_1.NOM:M] a signé.'],
|
||||
mapping: { '[PER_1.NOM:M]': 'Bob Martin' },
|
||||
legend: { PER: 'Personne', NOM: 'Nom de famille', M: 'Masculin' },
|
||||
});
|
||||
const s = new Anonymizer({ llm: failingLlm({ anonymizeBatch }), patterns: presets.swiss });
|
||||
const seed = { '[PER_1.NOM:M]': 'Alain JACCARD' };
|
||||
const r = await s.anonymizeChunks(['Bob Martin a signé.'], seed);
|
||||
const r = await s.anonymizeChunks(['Bob Martin a signé.'], { mapping: seed });
|
||||
expect(anonymizeBatch).toHaveBeenCalledTimes(1);
|
||||
expect(r.anon[0]).toContain('[PER_2.NOM:M]');
|
||||
expect(r.mapping['[PER_1.NOM:M]']).toBe('Alain JACCARD');
|
||||
expect(r.mapping['[PER_2.NOM:M]']).toBe('Bob Martin');
|
||||
// legend covers the abbreviations used in the anonymized chunk
|
||||
expect(r.legend).toMatchObject({ PER: 'Personne', NOM: 'Nom de famille', M: 'Masculin' });
|
||||
});
|
||||
|
||||
it('falls back to regex (structured ids) when the LLM is unavailable', async () => {
|
||||
const s = new Anonymizer({ llm: failingLlm(), patterns: presets.swiss });
|
||||
const r = await s.anonymizeChunks(['Contact : jean@exemple.ch'], {});
|
||||
const r = await s.anonymizeChunks(['Contact : jean@exemple.ch'], { mapping: {} });
|
||||
expect(r.anon[0]).not.toContain('jean@exemple.ch');
|
||||
expect(Object.values(r.mapping)).toContain('jean@exemple.ch');
|
||||
});
|
||||
@@ -121,4 +124,58 @@ describe('Anonymizer (Swiss preset)', () => {
|
||||
expect(r.mapping['[TICKET_1]']).toBe('JIRA-123');
|
||||
});
|
||||
});
|
||||
|
||||
describe('legend', () => {
|
||||
it('the regex fallback produces a French legend from tag meanings', async () => {
|
||||
const s = new Anonymizer({ patterns: presets.swiss });
|
||||
const r = await s.anonymize('Écris à jean@exemple.ch');
|
||||
expect(r.legend).toEqual({ EMAIL: 'Adresse e-mail' });
|
||||
});
|
||||
|
||||
it('backfills a missing legend entry from the built-in default (LLM omitted it)', async () => {
|
||||
const anonymize = vi.fn().mockResolvedValue({
|
||||
anon: 'Dossier de [PER_1.NOM:M]',
|
||||
mapping: { '[PER_1.NOM:M]': 'Alain Jaccard' },
|
||||
legend: {}, // model returned no legend
|
||||
});
|
||||
const s = new Anonymizer({ llm: failingLlm({ anonymize }), patterns: presets.swiss });
|
||||
const r = await s.anonymize('Dossier de Alain Jaccard');
|
||||
expect(r.legend).toEqual({ PER: 'Personne', NOM: 'Nom de famille', M: 'Masculin' });
|
||||
});
|
||||
|
||||
it('a custom tag with a meaning surfaces it in the legend', async () => {
|
||||
const s = new Anonymizer({
|
||||
patterns: [{ tag: 'TICKET', re: /\bJIRA-\d+\b/g, meaning: 'Ticket de suivi' }],
|
||||
});
|
||||
const r = await s.anonymize('Voir JIRA-123');
|
||||
expect(r.legend).toEqual({ TICKET: 'Ticket de suivi' });
|
||||
});
|
||||
});
|
||||
|
||||
describe('fail-closed (optional fallback)', () => {
|
||||
it('throws if neither an LLM nor patterns are provided', () => {
|
||||
expect(() => new Anonymizer({})).toThrowError(AnonymizationError);
|
||||
});
|
||||
|
||||
it('LLM-only: a provider failure throws AnonymizationError with the cause', async () => {
|
||||
const cause = new Error('LLM_HTTP_500');
|
||||
const s = new Anonymizer({ llm: failingLlm({ anonymize: vi.fn().mockRejectedValue(cause) }) });
|
||||
await expect(s.anonymize('Contact: jean@exemple.ch')).rejects.toBeInstanceOf(AnonymizationError);
|
||||
await expect(s.anonymize('Contact: jean@exemple.ch')).rejects.toMatchObject({ cause });
|
||||
});
|
||||
|
||||
it('LLM-only: bypasses the pre-filter so PII-free text still hits the provider', async () => {
|
||||
const anonymize = vi.fn().mockResolvedValue({ anon: 'INNER JOIN', mapping: {}, legend: {} });
|
||||
const s = new Anonymizer({ llm: failingLlm({ anonymize }) });
|
||||
await s.anonymize('Explique INNER JOIN'); // no structured PII, no name hint
|
||||
expect(anonymize).toHaveBeenCalledTimes(1);
|
||||
});
|
||||
|
||||
it('LLM-only: anonymizeChunks rejects with AnonymizationError on provider failure', async () => {
|
||||
const s = new Anonymizer({ llm: failingLlm() }); // anonymizeBatch rejects
|
||||
await expect(s.anonymizeChunks(['Bob Martin a signé.'], { mapping: {} })).rejects.toBeInstanceOf(
|
||||
AnonymizationError,
|
||||
);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@@ -25,7 +25,7 @@ describe('openAICompatibleProvider', () => {
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
const r = await openAICompatibleProvider(opts).anonymize('a@b.ch');
|
||||
expect(r).toEqual({ anon: '[EMAIL_1]', mapping: { '[EMAIL_1]': 'a@b.ch' } });
|
||||
expect(r).toEqual({ anon: '[EMAIL_1]', mapping: { '[EMAIL_1]': 'a@b.ch' }, legend: {} });
|
||||
|
||||
const [url, init] = fetchMock.mock.calls[0];
|
||||
expect(url).toBe('https://api.example.com/v1/chat/completions');
|
||||
@@ -35,6 +35,18 @@ describe('openAICompatibleProvider', () => {
|
||||
expect(init.headers.Authorization).toBe('Bearer k');
|
||||
});
|
||||
|
||||
it('parses the model legende into result.legend, keeping only string values', async () => {
|
||||
const fetchMock = mockFetchJson({
|
||||
texte_anonymise: '[PER_1.NOM:M]',
|
||||
mapping: { '[PER_1.NOM:M]': 'Alain Jaccard' },
|
||||
legende: { PER: 'Personne', NOM: 'Nom de famille', M: 'Masculin', BAD: 42 },
|
||||
});
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
const r = await openAICompatibleProvider(opts).anonymize('Alain Jaccard');
|
||||
expect(r.legend).toEqual({ PER: 'Personne', NOM: 'Nom de famille', M: 'Masculin' });
|
||||
});
|
||||
|
||||
it('anonymizeBatch() returns {segments, mapping} and includes used ids in the prompt', async () => {
|
||||
const fetchMock = mockFetchJson({ segments: ['[PER_2.NOM:M]'], mapping: { '[PER_2.NOM:M]': 'Bob' } });
|
||||
vi.stubGlobal('fetch', fetchMock);
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"target": "ES2021",
|
||||
"target": "ES2022",
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "Bundler",
|
||||
"lib": ["ES2021", "DOM"],
|
||||
"lib": ["ES2022", "DOM"],
|
||||
"declaration": true,
|
||||
"strict": true,
|
||||
"noUnusedLocals": true,
|
||||
|
||||
Reference in New Issue
Block a user