feat: multi-turn conversation support (0.4.0)
anonymizeTurn(text, session) threads a serializable AnonymizerSession
({mapping, legend, history}) so one entity keeps one id across a whole chat
(applyKnown reuse + usedIds + de-collision merge). Adds conversation()
in-memory wrapper and an optional LlmProvider.anonymizeInConversation(text, ctx)
for rich cross-turn context; providers without it fall back to the batch path.
openAICompatibleProvider gains includeMappingInContext (default false — only
send real values to a trusted anonymizer endpoint) + historyMaxTurns (default
10). Verified live vs Gemma 4: Nora stays PER_1 across 4 turns (name/email/AVS/
IBAN), Yanis = PER_2; 41 tests, 98% coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
18
CHANGELOG.md
18
CHANGELOG.md
@@ -4,6 +4,24 @@ All notable changes to this project are documented here. The format is based on
|
|||||||
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
|
||||||
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [0.4.0] - Unreleased
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- **Multi-turn conversation support.** New `Anonymizer.anonymizeTurn(text, session?) → { anon, mapping,
|
||||||
|
legend, session }` keeps one stable id per entity across a whole chat: it seeds each turn with a running,
|
||||||
|
serializable `AnonymizerSession` (`{ mapping, legend, history }`), reuses known values via `applyKnown`,
|
||||||
|
tells the model which ids are taken, and de-collides new ones. Persist the returned `session` and pass it
|
||||||
|
back next turn.
|
||||||
|
- `Anonymizer.conversation(initial?)` — a stateful in-memory wrapper (`anonymize`, `deanonymize`,
|
||||||
|
`session()`) over `anonymizeTurn`.
|
||||||
|
- Optional `LlmProvider.anonymizeInConversation(text, ctx)` — providers can use prior context (anonymized
|
||||||
|
`history`, `legend`, `usedIds`, and optionally `mapping`) for better cross-turn coreference/attribution.
|
||||||
|
`openAICompatibleProvider` implements it; providers that don't fall back to the batch path automatically.
|
||||||
|
- `openAICompatibleProvider` options: `includeMappingInContext` (**default false** — only send real values
|
||||||
|
to a _trusted_ anonymizer endpoint) and `historyMaxTurns` via `AnonymizerConfig` (default 10).
|
||||||
|
- Exported the `AnonymizerSession` type.
|
||||||
|
|
||||||
## [0.3.1] - Unreleased
|
## [0.3.1] - Unreleased
|
||||||
|
|
||||||
### Changed
|
### Changed
|
||||||
|
|||||||
36
README.md
36
README.md
@@ -108,6 +108,42 @@ const { anon, mapping, legend } = await anonymizer.anonymizeChunks(retrievedChun
|
|||||||
// pass `mapping` to deanonymize()/makeStreamDeanonymizer(), and `legend` to the downstream LLM.
|
// pass `mapping` to deanonymize()/makeStreamDeanonymizer(), and `legend` to the downstream LLM.
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Multi-turn conversations
|
||||||
|
|
||||||
|
In a chat, anonymizing each message independently would renumber entities every turn (Adil could become
|
||||||
|
`PER_1` in turn 2 while Oussama was `PER_1` in turn 1). `anonymizeTurn` threads a serializable
|
||||||
|
`AnonymizerSession` so **one entity keeps one id across the whole conversation**:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
let session; // persist this per conversation (Redis/DB); pass it back each turn
|
||||||
|
for (const message of userTurns) {
|
||||||
|
const turn = await anonymizer.anonymizeTurn(message, session);
|
||||||
|
session = turn.session; // { mapping, legend, history } — carries coreference forward
|
||||||
|
send(turn.anon, turn.legend); // → the downstream chatbot (placeholders only)
|
||||||
|
}
|
||||||
|
// restore the bot's placeholder-bearing reply for the user:
|
||||||
|
anonymizer.deanonymize(botReply, session.mapping);
|
||||||
|
```
|
||||||
|
|
||||||
|
Or the stateful wrapper for in-memory use:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const conv = anonymizer.conversation();
|
||||||
|
await conv.anonymize('Bonjour, je suis Oussama'); // → Oussama = [PER_1…]
|
||||||
|
await conv.anonymize('Mon amie Adil …'); // → Oussama stays [PER_1…], Adil = [PER_2…]
|
||||||
|
conv.deanonymize(botReply);
|
||||||
|
```
|
||||||
|
|
||||||
|
Under the hood: known values are re-substituted locally (`applyKnown`), the provider is told which ids are
|
||||||
|
taken, and new entities are de-collided into the session — so numbering never drifts.
|
||||||
|
|
||||||
|
**Trust boundary & the `mapping`.** The real boundary is "keep PII out of the _downstream chatbot_" — it
|
||||||
|
only ever receives placeholders + `legend`. Your **anonymizer** endpoint already sees the cleartext it's
|
||||||
|
asked to anonymize, so if (and only if) it's a _trusted_ processor you may give it richer context — set
|
||||||
|
`includeMappingInContext: true` on `openAICompatibleProvider` to also send the `mapping` for maximum
|
||||||
|
cross-turn accuracy. It's **off by default**; keep it off for third-party endpoints you don't trust with
|
||||||
|
raw values.
|
||||||
|
|
||||||
## Configuration
|
## Configuration
|
||||||
|
|
||||||
```ts
|
```ts
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
{
|
{
|
||||||
"name": "@mobiletic/anonymizer",
|
"name": "@mobiletic/anonymizer",
|
||||||
"version": "0.3.1",
|
"version": "0.4.0",
|
||||||
"description": "Framework-agnostic PII anonymization & pseudonymization: pluggable LLM detection for free-form PII (names, addresses) with a deterministic regex fallback, deterministic coreference, and streaming-safe de-anonymization.",
|
"description": "Framework-agnostic PII anonymization & pseudonymization: pluggable LLM detection for free-form PII (names, addresses) with a deterministic regex fallback, deterministic coreference, and streaming-safe de-anonymization.",
|
||||||
"license": "MIT",
|
"license": "MIT",
|
||||||
"author": "Mobiletic",
|
"author": "Mobiletic",
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ import { AnonymizationError } from './errors.js';
|
|||||||
import {
|
import {
|
||||||
type AnonymizationResult,
|
type AnonymizationResult,
|
||||||
type AnonymizerConfig,
|
type AnonymizerConfig,
|
||||||
|
type AnonymizerSession,
|
||||||
type LlmProvider,
|
type LlmProvider,
|
||||||
type Logger,
|
type Logger,
|
||||||
type StreamDeanonymizer,
|
type StreamDeanonymizer,
|
||||||
@@ -73,12 +74,14 @@ export class Anonymizer {
|
|||||||
private readonly fallback?: RegexFallback;
|
private readonly fallback?: RegexFallback;
|
||||||
private readonly nameHint: RegExp;
|
private readonly nameHint: RegExp;
|
||||||
private readonly logger: Logger;
|
private readonly logger: Logger;
|
||||||
|
private readonly historyMaxTurns: number;
|
||||||
|
|
||||||
constructor(config: AnonymizerConfig = {}) {
|
constructor(config: AnonymizerConfig = {}) {
|
||||||
this.llm = config.llm;
|
this.llm = config.llm;
|
||||||
this.fallback = config.patterns ? new RegexFallback(config.patterns) : undefined;
|
this.fallback = config.patterns ? new RegexFallback(config.patterns) : undefined;
|
||||||
this.nameHint = stateless(config.nameHint ?? DEFAULT_NAME_HINT);
|
this.nameHint = stateless(config.nameHint ?? DEFAULT_NAME_HINT);
|
||||||
this.logger = config.logger ?? NOOP_LOGGER;
|
this.logger = config.logger ?? NOOP_LOGGER;
|
||||||
|
this.historyMaxTurns = config.historyMaxTurns ?? 10;
|
||||||
if (!this.llm && !this.fallback) {
|
if (!this.llm && !this.fallback) {
|
||||||
throw new AnonymizationError('Anonymizer requires an LlmProvider, regex patterns, or both');
|
throw new AnonymizationError('Anonymizer requires an LlmProvider, regex patterns, or both');
|
||||||
}
|
}
|
||||||
@@ -169,6 +172,109 @@ export class Anonymizer {
|
|||||||
return this.fallbackChunks(seeded, seed.mapping, seedLegend);
|
return this.fallbackChunks(seeded, seed.mapping, seedLegend);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Anonymize ONE conversation turn, consistently with a running
|
||||||
|
* {@link AnonymizerSession}, so an entity keeps the same id across the whole
|
||||||
|
* chat. Returns the anonymized turn + the updated session (persist it and pass
|
||||||
|
* it back next turn). Strategy per turn:
|
||||||
|
* 1. `applyKnown` — swap already-known values to their placeholders (id reuse,
|
||||||
|
* no value leaves for those spans).
|
||||||
|
* 2. Detect: if the provider supports {@link LlmProvider.anonymizeInConversation}
|
||||||
|
* it gets prior context (history/legend/usedIds, and the mapping only if the
|
||||||
|
* provider opted in); otherwise fall back to the batched path with `usedIds`
|
||||||
|
* (still collision-free). No LLM → regex only.
|
||||||
|
* 3. `mergeMappings` de-collides into the session, `validate` anti-leak, and the
|
||||||
|
* anonymized turn is appended to `history` (capped to `historyMaxTurns`).
|
||||||
|
* On failure with no regex fallback → throws {@link AnonymizationError} (fail-closed).
|
||||||
|
*/
|
||||||
|
async anonymizeTurn(
|
||||||
|
text: string,
|
||||||
|
session?: AnonymizerSession,
|
||||||
|
): Promise<{
|
||||||
|
anon: string;
|
||||||
|
mapping: Record<string, string>;
|
||||||
|
legend: Record<string, string>;
|
||||||
|
session: AnonymizerSession;
|
||||||
|
}> {
|
||||||
|
const prev: AnonymizerSession = session ?? { mapping: {}, legend: {}, history: [] };
|
||||||
|
const seeded = this.applyKnown(text, prev.mapping);
|
||||||
|
|
||||||
|
let turn: AnonymizationResult;
|
||||||
|
if (this.hasLlm()) {
|
||||||
|
try {
|
||||||
|
if (this.llm!.anonymizeInConversation) {
|
||||||
|
turn = await this.llm!.anonymizeInConversation(seeded, {
|
||||||
|
history: prev.history,
|
||||||
|
legend: prev.legend,
|
||||||
|
usedIds: Object.keys(prev.mapping),
|
||||||
|
mapping: prev.mapping, // provider decides whether to actually transmit it
|
||||||
|
});
|
||||||
|
} else {
|
||||||
|
const b = await this.llm!.anonymizeBatch([seeded], Object.keys(prev.mapping));
|
||||||
|
if (b.segments.length !== 1) throw new Error('SEGMENT_COUNT_MISMATCH');
|
||||||
|
turn = { anon: b.segments[0], mapping: b.mapping, legend: b.legend };
|
||||||
|
}
|
||||||
|
return this.commitTurn(prev, this.mergeAndCheck(prev.mapping, turn));
|
||||||
|
} catch (err) {
|
||||||
|
if (!this.fallback) {
|
||||||
|
throw new AnonymizationError(`anonymizeTurn failed: ${(err as Error).message}`, { cause: err });
|
||||||
|
}
|
||||||
|
this.logger.warn(`Conversation turn → regex fallback: ${(err as Error).message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return this.commitTurn(prev, this.mergeAndCheck(prev.mapping, this.fallback!.anonymize(seeded)));
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Merge a turn's result into the running mapping (de-collision) + anti-leak check. */
|
||||||
|
private mergeAndCheck(
|
||||||
|
seedMapping: Record<string, string>,
|
||||||
|
result: AnonymizationResult,
|
||||||
|
): { anon: string; mapping: Record<string, string>; legend: Record<string, string> } {
|
||||||
|
const { mapping, rename } = this.mergeMappings(seedMapping, result.mapping);
|
||||||
|
const anon = this.applyRename(result.anon, rename);
|
||||||
|
for (const ph of anon.match(PLACEHOLDER_RE) ?? []) {
|
||||||
|
if (!(ph in mapping)) throw new Error(`unmapped placeholder ${ph}`);
|
||||||
|
}
|
||||||
|
return { anon, mapping, legend: result.legend };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Fold a merged turn into the session: cap history, rebuild the cumulative legend. */
|
||||||
|
private commitTurn(
|
||||||
|
prev: AnonymizerSession,
|
||||||
|
turn: { anon: string; mapping: Record<string, string>; legend: Record<string, string> },
|
||||||
|
): {
|
||||||
|
anon: string;
|
||||||
|
mapping: Record<string, string>;
|
||||||
|
legend: Record<string, string>;
|
||||||
|
session: AnonymizerSession;
|
||||||
|
} {
|
||||||
|
const history = [...prev.history, turn.anon].slice(-this.historyMaxTurns);
|
||||||
|
const legend = this.buildLegend(history.join('\n'), { ...prev.legend, ...turn.legend });
|
||||||
|
const nextSession: AnonymizerSession = { mapping: turn.mapping, legend, history };
|
||||||
|
return { anon: turn.anon, mapping: turn.mapping, legend, session: nextSession };
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Stateful convenience wrapper over {@link anonymizeTurn} for in-memory use.
|
||||||
|
* Holds the evolving session so callers just do `await conv.anonymize(text)`.
|
||||||
|
*/
|
||||||
|
conversation(initial?: AnonymizerSession): {
|
||||||
|
anonymize: (text: string) => Promise<AnonymizationResult>;
|
||||||
|
deanonymize: (text: string) => string;
|
||||||
|
session: () => AnonymizerSession;
|
||||||
|
} {
|
||||||
|
let session: AnonymizerSession = initial ?? { mapping: {}, legend: {}, history: [] };
|
||||||
|
return {
|
||||||
|
anonymize: async (text: string) => {
|
||||||
|
const r = await this.anonymizeTurn(text, session);
|
||||||
|
session = r.session;
|
||||||
|
return { anon: r.anon, mapping: r.mapping, legend: r.legend };
|
||||||
|
},
|
||||||
|
deanonymize: (text: string) => this.deanonymize(text, session.mapping),
|
||||||
|
session: () => session,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
/** Replace known values (seed) by their placeholder, longest values first. */
|
/** Replace known values (seed) by their placeholder, longest values first. */
|
||||||
private applyKnown(text: string, seed: Record<string, string>): string {
|
private applyKnown(text: string, seed: Record<string, string>): string {
|
||||||
let out = text;
|
let out = text;
|
||||||
|
|||||||
@@ -11,6 +11,7 @@ export {
|
|||||||
PLACEHOLDER_RE,
|
PLACEHOLDER_RE,
|
||||||
type AnonymizationResult,
|
type AnonymizationResult,
|
||||||
type AnonymizerConfig,
|
type AnonymizerConfig,
|
||||||
|
type AnonymizerSession,
|
||||||
type LlmProvider,
|
type LlmProvider,
|
||||||
type PatternDef,
|
type PatternDef,
|
||||||
type Logger,
|
type Logger,
|
||||||
|
|||||||
@@ -64,6 +64,15 @@ export interface OpenAICompatibleOptions {
|
|||||||
* responses are parsed leniently (markdown fences / surrounding prose tolerated).
|
* responses are parsed leniently (markdown fences / surrounding prose tolerated).
|
||||||
*/
|
*/
|
||||||
responseFormat?: Record<string, unknown>;
|
responseFormat?: Record<string, unknown>;
|
||||||
|
/**
|
||||||
|
* Include the secret `mapping` (placeholder → real value) in the conversation
|
||||||
|
* context during {@link LlmProvider.anonymizeInConversation}. **Default false.**
|
||||||
|
* Enable ONLY when this endpoint is a trusted processor — it sends REAL values
|
||||||
|
* to the model. (The anonymizer already receives the cleartext being
|
||||||
|
* anonymized, so a trusted endpoint like a Swiss/self-hosted deployment sees no
|
||||||
|
* more than it already does; the downstream chatbot never receives the mapping.)
|
||||||
|
*/
|
||||||
|
includeMappingInContext?: boolean;
|
||||||
/** Override the system prompt (e.g. for another language or regulation). */
|
/** Override the system prompt (e.g. for another language or regulation). */
|
||||||
systemPrompt?: string;
|
systemPrompt?: string;
|
||||||
/** Extra instructions appended to the system prompt in batch mode. */
|
/** Extra instructions appended to the system prompt in batch mode. */
|
||||||
@@ -115,6 +124,42 @@ const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
|||||||
'\n- SORTIE STRICTEMENT JSON : {"segments": ["…anonymisé…"], "mapping": {"[PER_1.NOM:M]": "…"}, "legende": {"PER": "Personne"}}.' +
|
'\n- SORTIE STRICTEMENT JSON : {"segments": ["…anonymisé…"], "mapping": {"[PER_1.NOM:M]": "…"}, "legende": {"PER": "Personne"}}.' +
|
||||||
'\n- "segments" DOIT avoir la même longueur et le même ordre que l’entrée ; "mapping" ne contient que les NOUVELLES entités ; "legende" décrit toutes les abréviations utilisées.';
|
'\n- "segments" DOIT avoir la même longueur et le même ordre que l’entrée ; "mapping" ne contient que les NOUVELLES entités ; "legende" décrit toutes les abréviations utilisées.';
|
||||||
|
|
||||||
|
/** Prior-conversation context block appended to the system prompt for a turn. */
|
||||||
|
function conversationContext(
|
||||||
|
ctx: {
|
||||||
|
history: string[];
|
||||||
|
legend: Record<string, string>;
|
||||||
|
usedIds: string[];
|
||||||
|
mapping?: Record<string, string>;
|
||||||
|
},
|
||||||
|
includeMapping: boolean,
|
||||||
|
): string {
|
||||||
|
const lines = ['', 'CONTEXTE DE LA CONVERSATION (déjà pseudonymisé) :'];
|
||||||
|
if (ctx.history.length) {
|
||||||
|
lines.push('- Tours précédents :', ...ctx.history.map((h, i) => ` [${i + 1}] ${h}`));
|
||||||
|
}
|
||||||
|
if (Object.keys(ctx.legend).length) {
|
||||||
|
lines.push(`- Légende connue : ${JSON.stringify(ctx.legend)}`);
|
||||||
|
}
|
||||||
|
if (ctx.usedIds.length) {
|
||||||
|
lines.push(
|
||||||
|
`- Identifiants DÉJÀ attribués (réutilise-les pour les MÊMES entités, n'en réattribue AUCUN à une autre valeur) : ${ctx.usedIds.join(', ')}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
if (includeMapping && ctx.mapping && Object.keys(ctx.mapping).length) {
|
||||||
|
const pairs = Object.entries(ctx.mapping)
|
||||||
|
.map(([ph, v]) => `${ph} = ${v}`)
|
||||||
|
.join(' ; ');
|
||||||
|
lines.push(
|
||||||
|
`- Correspondances connues (réutilise le MÊME placeholder si la valeur réapparaît) : ${pairs}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
lines.push(
|
||||||
|
'Anonymise le MESSAGE suivant en gardant EXACTEMENT la même convention et les mêmes identifiants pour les entités déjà vues ; "mapping" ne contient que les NOUVELLES entités.',
|
||||||
|
);
|
||||||
|
return '\n\n' + lines.join('\n');
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Build an {@link LlmProvider} backed by any OpenAI-compatible Chat Completions
|
* Build an {@link LlmProvider} backed by any OpenAI-compatible Chat Completions
|
||||||
* endpoint (OpenAI, Infomaniak, vLLM, Ollama, …). Uses `temperature: 0` for
|
* endpoint (OpenAI, Infomaniak, vLLM, Ollama, …). Uses `temperature: 0` for
|
||||||
@@ -128,6 +173,7 @@ export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProv
|
|||||||
const retries = Math.max(0, opts.retries ?? 1);
|
const retries = Math.max(0, opts.retries ?? 1);
|
||||||
const retryDelayMs = opts.retryDelayMs ?? 250;
|
const retryDelayMs = opts.retryDelayMs ?? 250;
|
||||||
const responseFormat = opts.responseFormat;
|
const responseFormat = opts.responseFormat;
|
||||||
|
const includeMappingInContext = opts.includeMappingInContext ?? false;
|
||||||
const systemPrompt = opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
|
const systemPrompt = opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
|
||||||
const batchInstructions = opts.batchInstructions ?? DEFAULT_BATCH_INSTRUCTIONS;
|
const batchInstructions = opts.batchInstructions ?? DEFAULT_BATCH_INSTRUCTIONS;
|
||||||
|
|
||||||
@@ -206,6 +252,24 @@ export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProv
|
|||||||
return { anon: parsed.texte_anonymise, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
return { anon: parsed.texte_anonymise, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||||||
},
|
},
|
||||||
|
|
||||||
|
async anonymizeInConversation(text, ctx): Promise<AnonymizationResult> {
|
||||||
|
const system = systemPrompt + conversationContext(ctx, includeMappingInContext);
|
||||||
|
const content = await chat(system, text);
|
||||||
|
const parsed = extractJson(content) as {
|
||||||
|
texte_anonymise?: string;
|
||||||
|
mapping?: Record<string, string>;
|
||||||
|
legende?: Record<string, string>;
|
||||||
|
};
|
||||||
|
if (
|
||||||
|
typeof parsed.texte_anonymise !== 'string' ||
|
||||||
|
typeof parsed.mapping !== 'object' ||
|
||||||
|
!parsed.mapping
|
||||||
|
) {
|
||||||
|
throw new Error('LLM_BAD_SHAPE');
|
||||||
|
}
|
||||||
|
return { anon: parsed.texte_anonymise, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||||||
|
},
|
||||||
|
|
||||||
async anonymizeBatch(
|
async anonymizeBatch(
|
||||||
texts: string[],
|
texts: string[],
|
||||||
usedIds: string[],
|
usedIds: string[],
|
||||||
|
|||||||
39
src/types.ts
39
src/types.ts
@@ -40,6 +40,39 @@ export interface LlmProvider {
|
|||||||
texts: string[],
|
texts: string[],
|
||||||
usedIds: string[],
|
usedIds: string[],
|
||||||
): Promise<{ segments: string[]; mapping: Record<string, string>; legend: Record<string, string> }>;
|
): Promise<{ segments: string[]; mapping: Record<string, string>; legend: Record<string, string> }>;
|
||||||
|
/**
|
||||||
|
* OPTIONAL — anonymize one conversation turn with prior context so entity ids
|
||||||
|
* stay stable across turns. `ctx.history` is the (already anonymized) prior
|
||||||
|
* turns, `ctx.legend` the accumulated abbreviations, `ctx.usedIds` the ids
|
||||||
|
* already assigned, and `ctx.mapping` the placeholder→value table — the latter
|
||||||
|
* is only passed when the caller has opted in AND the provider is trusted with
|
||||||
|
* cleartext (it already sees the message being anonymized). Providers that omit
|
||||||
|
* this method still work: {@link Anonymizer.anonymizeTurn} falls back to the
|
||||||
|
* batch path.
|
||||||
|
*/
|
||||||
|
anonymizeInConversation?(
|
||||||
|
text: string,
|
||||||
|
ctx: {
|
||||||
|
history: string[];
|
||||||
|
legend: Record<string, string>;
|
||||||
|
usedIds: string[];
|
||||||
|
mapping?: Record<string, string>;
|
||||||
|
},
|
||||||
|
): Promise<AnonymizationResult>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Serializable state for a multi-turn conversation, persisted by the caller
|
||||||
|
* (e.g. in a session store) and threaded through {@link Anonymizer.anonymizeTurn}
|
||||||
|
* so one entity keeps one id across the whole chat.
|
||||||
|
*/
|
||||||
|
export interface AnonymizerSession {
|
||||||
|
/** Placeholder → original value across the whole conversation. SECRET — keep it on your side. */
|
||||||
|
mapping: Record<string, string>;
|
||||||
|
/** Abbreviation → meaning across the whole conversation. Non-secret. */
|
||||||
|
legend: Record<string, string>;
|
||||||
|
/** Anonymized prior turns (most recent last), capped by `historyMaxTurns`. */
|
||||||
|
history: string[];
|
||||||
}
|
}
|
||||||
|
|
||||||
/** A structured-PII detector: a tag (e.g. `EMAIL`) and the global regex that finds it. */
|
/** A structured-PII detector: a tag (e.g. `EMAIL`) and the global regex that finds it. */
|
||||||
@@ -84,6 +117,12 @@ export interface AnonymizerConfig {
|
|||||||
nameHint?: RegExp;
|
nameHint?: RegExp;
|
||||||
/** Where fallback/diagnostic warnings go. Defaults to a no-op. */
|
/** Where fallback/diagnostic warnings go. Defaults to a no-op. */
|
||||||
logger?: Logger;
|
logger?: Logger;
|
||||||
|
/**
|
||||||
|
* Max number of prior anonymized turns kept in an {@link AnonymizerSession}
|
||||||
|
* history (and thus fed back to the provider). Default 10. Caps token/latency
|
||||||
|
* growth over long conversations.
|
||||||
|
*/
|
||||||
|
historyMaxTurns?: number;
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
|
|||||||
@@ -1,5 +1,11 @@
|
|||||||
import { describe, it, expect, vi } from 'vitest';
|
import { describe, it, expect, vi } from 'vitest';
|
||||||
import { Anonymizer, AnonymizationError, presets, type LlmProvider } from '../src/index.js';
|
import {
|
||||||
|
Anonymizer,
|
||||||
|
AnonymizationError,
|
||||||
|
presets,
|
||||||
|
type LlmProvider,
|
||||||
|
type AnonymizerSession,
|
||||||
|
} from '../src/index.js';
|
||||||
|
|
||||||
/** An LLM provider that is configured but always fails → forces the regex fallback. */
|
/** An LLM provider that is configured but always fails → forces the regex fallback. */
|
||||||
const failingLlm = (overrides: Partial<LlmProvider> = {}): LlmProvider => ({
|
const failingLlm = (overrides: Partial<LlmProvider> = {}): LlmProvider => ({
|
||||||
@@ -179,3 +185,113 @@ describe('Anonymizer (Swiss preset)', () => {
|
|||||||
});
|
});
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe('multi-turn conversation (anonymizeTurn)', () => {
|
||||||
|
it('keeps stable ids across turns and accumulates the session', async () => {
|
||||||
|
const anonymizeInConversation = vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValueOnce({
|
||||||
|
anon: 'Je suis [PER_1.NOM:F]',
|
||||||
|
mapping: { '[PER_1.NOM:F]': 'Nora Steiner' },
|
||||||
|
legend: { PER: 'Personne', NOM: 'Nom de famille', F: 'Féminin' },
|
||||||
|
})
|
||||||
|
.mockResolvedValueOnce({
|
||||||
|
anon: 'Ma collègue [PER_2.NOM:F] a aussi un souci',
|
||||||
|
mapping: { '[PER_2.NOM:F]': 'Yanis Berger' },
|
||||||
|
legend: { PER: 'Personne', NOM: 'Nom de famille', F: 'Féminin' },
|
||||||
|
});
|
||||||
|
const llm = failingLlm({ anonymizeInConversation });
|
||||||
|
const a = new Anonymizer({ llm });
|
||||||
|
|
||||||
|
const t1 = await a.anonymizeTurn('Je suis Nora Steiner');
|
||||||
|
expect(t1.mapping).toEqual({ '[PER_1.NOM:F]': 'Nora Steiner' });
|
||||||
|
|
||||||
|
const t2 = await a.anonymizeTurn('Ma collègue Yanis Berger a aussi un souci', t1.session);
|
||||||
|
expect(t2.mapping['[PER_1.NOM:F]']).toBe('Nora Steiner'); // stable across turns
|
||||||
|
expect(t2.mapping['[PER_2.NOM:F]']).toBe('Yanis Berger'); // new person, no collision
|
||||||
|
expect(t2.session.history).toHaveLength(2);
|
||||||
|
// the second call was told PER_1 is already used
|
||||||
|
expect(anonymizeInConversation.mock.calls[1][1].usedIds).toContain('[PER_1.NOM:F]');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('conversation() wrapper threads the session and de-anonymizes', async () => {
|
||||||
|
const anonymizeInConversation = vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValueOnce({
|
||||||
|
anon: '[PER_1.PRENOM:M]',
|
||||||
|
mapping: { '[PER_1.PRENOM:M]': 'Idris' },
|
||||||
|
legend: {},
|
||||||
|
})
|
||||||
|
.mockResolvedValueOnce({
|
||||||
|
anon: '[PER_1.PRENOM:M] et [PER_2.PRENOM:F]',
|
||||||
|
mapping: { '[PER_2.PRENOM:F]': 'Lina' },
|
||||||
|
legend: {},
|
||||||
|
});
|
||||||
|
const conv = new Anonymizer({ llm: failingLlm({ anonymizeInConversation }) }).conversation();
|
||||||
|
await conv.anonymize('Idris');
|
||||||
|
await conv.anonymize('Idris et Lina');
|
||||||
|
expect(conv.session().mapping).toEqual({ '[PER_1.PRENOM:M]': 'Idris', '[PER_2.PRENOM:F]': 'Lina' });
|
||||||
|
expect(conv.deanonymize('[PER_1.PRENOM:M]')).toBe('Idris');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to the batch path when the provider has no anonymizeInConversation', async () => {
|
||||||
|
const anonymizeBatch = vi.fn().mockResolvedValue({
|
||||||
|
segments: ['[PER_1.NOM:M]'],
|
||||||
|
mapping: { '[PER_1.NOM:M]': 'Bruno Keller' },
|
||||||
|
legend: {},
|
||||||
|
});
|
||||||
|
const a = new Anonymizer({ llm: failingLlm({ anonymizeBatch }) });
|
||||||
|
const t = await a.anonymizeTurn('Bruno Keller');
|
||||||
|
expect(anonymizeBatch).toHaveBeenCalledWith(['Bruno Keller'], []);
|
||||||
|
expect(t.mapping['[PER_1.NOM:M]']).toBe('Bruno Keller');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('realistic e-learning chat: stable coreference + attribute attribution + round-trips', async () => {
|
||||||
|
const turns = [
|
||||||
|
{
|
||||||
|
text: "Bonjour, je suis Nora Steiner, inscrite à la formation « Assistante médicale ». Je n'ai pas reçu ma convocation.",
|
||||||
|
anon: "Bonjour, je suis [PER_1.PRENOM:F] [PER_1.NOM:F], inscrite à la formation « Assistante médicale ». Je n'ai pas reçu ma convocation.",
|
||||||
|
mapping: { '[PER_1.PRENOM:F]': 'Nora', '[PER_1.NOM:F]': 'Steiner' },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
text: 'Mon e-mail est nora.steiner@hotmail.ch et mon numéro AVS 756.2233.4455.66 au cas où.',
|
||||||
|
anon: 'Mon e-mail est [PER_1.EMAIL:F] et mon numéro AVS [PER_1.AVS:F] au cas où.',
|
||||||
|
mapping: { '[PER_1.EMAIL:F]': 'nora.steiner@hotmail.ch', '[PER_1.AVS:F]': '756.2233.4455.66' },
|
||||||
|
},
|
||||||
|
{
|
||||||
|
text: "En fait c'est aussi pour ma collègue Yanis Berger — elle veut s'inscrire, son tél. 078 111 22 33.",
|
||||||
|
anon: "En fait c'est aussi pour ma collègue [PER_2.PRENOM:F] [PER_2.NOM:F] — elle veut s'inscrire, son tél. [PER_2.TELEPHONE:F].",
|
||||||
|
mapping: {
|
||||||
|
'[PER_2.PRENOM:F]': 'Yanis',
|
||||||
|
'[PER_2.NOM:F]': 'Berger',
|
||||||
|
'[PER_2.TELEPHONE:F]': '078 111 22 33',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
text: 'Le paiement se fera depuis mon IBAN CH88 0900 0000 1234 5678 9. Merci !',
|
||||||
|
anon: 'Le paiement se fera depuis mon IBAN [PER_1.IBAN:F]. Merci !',
|
||||||
|
mapping: { '[PER_1.IBAN:F]': 'CH88 0900 0000 1234 5678 9' },
|
||||||
|
},
|
||||||
|
];
|
||||||
|
const anonymizeInConversation = vi.fn();
|
||||||
|
for (const t of turns) {
|
||||||
|
anonymizeInConversation.mockResolvedValueOnce({ anon: t.anon, mapping: t.mapping, legend: {} });
|
||||||
|
}
|
||||||
|
const a = new Anonymizer({ llm: failingLlm({ anonymizeInConversation }) });
|
||||||
|
|
||||||
|
let session: AnonymizerSession = { mapping: {}, legend: {}, history: [] };
|
||||||
|
for (const t of turns) {
|
||||||
|
const r = await a.anonymizeTurn(t.text, session);
|
||||||
|
session = r.session;
|
||||||
|
// each turn restores to the exact original
|
||||||
|
expect(a.deanonymize(r.anon, r.mapping)).toBe(t.text);
|
||||||
|
}
|
||||||
|
// Nora = PER_1 throughout (name + email + AVS + IBAN attached to her); Yanis = PER_2.
|
||||||
|
expect(session.mapping['[PER_1.PRENOM:F]']).toBe('Nora');
|
||||||
|
expect(session.mapping['[PER_1.EMAIL:F]']).toBe('nora.steiner@hotmail.ch');
|
||||||
|
expect(session.mapping['[PER_1.IBAN:F]']).toBe('CH88 0900 0000 1234 5678 9');
|
||||||
|
expect(session.mapping['[PER_2.PRENOM:F]']).toBe('Yanis');
|
||||||
|
expect(session.mapping['[PER_2.TELEPHONE:F]']).toBe('078 111 22 33');
|
||||||
|
expect(session.history).toHaveLength(4);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|||||||
@@ -79,6 +79,39 @@ describe('openAICompatibleProvider', () => {
|
|||||||
expect(system).toContain('[PER_1.NOM:M]');
|
expect(system).toContain('[PER_1.NOM:M]');
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it('anonymizeInConversation includes used-ids + history, and the mapping ONLY when opted in', async () => {
|
||||||
|
const content = JSON.stringify({
|
||||||
|
texte_anonymise: '[PER_2.NOM:F]',
|
||||||
|
mapping: { '[PER_2.NOM:F]': 'X' },
|
||||||
|
legende: {},
|
||||||
|
});
|
||||||
|
const fetchMock = vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValue({ ok: true, json: async () => ({ choices: [{ message: { content } }] }) });
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
const ctx = {
|
||||||
|
history: ['Je suis [PER_1.NOM:F]'],
|
||||||
|
legend: { PER: 'Personne' },
|
||||||
|
usedIds: ['[PER_1.NOM:F]'],
|
||||||
|
mapping: { '[PER_1.NOM:F]': 'Nora Steiner' },
|
||||||
|
};
|
||||||
|
|
||||||
|
// default: real values (mapping) are NOT sent
|
||||||
|
await openAICompatibleProvider(opts).anonymizeInConversation!('x', ctx);
|
||||||
|
let system = JSON.parse(fetchMock.mock.calls[0][1].body).messages[0].content;
|
||||||
|
expect(system).toContain('[PER_1.NOM:F]'); // used-ids present
|
||||||
|
expect(system).toContain('Je suis [PER_1.NOM:F]'); // anonymized history present
|
||||||
|
expect(system).not.toContain('Nora Steiner'); // real value withheld by default
|
||||||
|
|
||||||
|
// opted in: the mapping (real value) is included
|
||||||
|
await openAICompatibleProvider({ ...opts, includeMappingInContext: true }).anonymizeInConversation!(
|
||||||
|
'x',
|
||||||
|
ctx,
|
||||||
|
);
|
||||||
|
system = JSON.parse(fetchMock.mock.calls[1][1].body).messages[0].content;
|
||||||
|
expect(system).toContain('Nora Steiner');
|
||||||
|
});
|
||||||
|
|
||||||
it('throws on a non-OK HTTP status (no retry when retries: 0)', async () => {
|
it('throws on a non-OK HTTP status (no retry when retries: 0)', async () => {
|
||||||
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 500 });
|
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 500 });
|
||||||
vi.stubGlobal('fetch', fetchMock);
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
|||||||
Reference in New Issue
Block a user