5ed8001101
anonymizeTurn(text, session) threads a serializable AnonymizerSession
({mapping, legend, history}) so one entity keeps one id across a whole chat
(applyKnown reuse + usedIds + de-collision merge). Adds conversation()
in-memory wrapper and an optional LlmProvider.anonymizeInConversation(text, ctx)
for rich cross-turn context; providers without it fall back to the batch path.
openAICompatibleProvider gains includeMappingInContext (default false — only
send real values to a trusted anonymizer endpoint) + historyMaxTurns (default
10). Verified live vs Gemma 4: Nora stays PER_1 across 4 turns (name/email/AVS/
IBAN), Yanis = PER_2; 41 tests, 98% coverage.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
291 lines
13 KiB
TypeScript
291 lines
13 KiB
TypeScript
import type { AnonymizationResult, LlmProvider } from '../types.js';
|
||
|
||
/** Default nLPD pseudonymization prompt (French). Override for other locales/regulations. */
|
||
export const DEFAULT_SYSTEM_PROMPT = [
|
||
'Tu es un moteur de pseudonymisation conforme à la nLPD suisse.',
|
||
'Identifie UNIQUEMENT les données personnelles (identifiants directs et indirects)',
|
||
'et remplace-les par des placeholders. Ne touche à RIEN d’autre.',
|
||
'',
|
||
'FORMAT: [ENTITE_ID.ATTRIBUT:CONTEXTE]',
|
||
'Entités: PER (personne), ORG (organisation), LOC (lieu autonome).',
|
||
'Attributs PER: NOM, PRENOM, DATE_NAISSANCE, AGE, ADRESSE, EMAIL, TELEPHONE, AVS, IBAN, NSS.',
|
||
'Contexte = indice non-identifiant utile au raisonnement:',
|
||
' NOM:M|F|U · DATE_NAISSANCE:<année> · AGE:Mineur|Adulte',
|
||
' ADRESSE:Lieu|Rue|Ville|NPA|Pays · ORG:Entreprise|Ecole',
|
||
'',
|
||
'ABRÉVIATIONS DYNAMIQUES:',
|
||
'- Si tu rencontres un TYPE d’entité, un ATTRIBUT ou un CONTEXTE non listé ci-dessus,',
|
||
' INVENTE une abréviation COURTE en MAJUSCULES (lettres A–Z et "_" uniquement, ex. PER, NOM, M),',
|
||
' suivant la même convention, et réutilise-la de façon cohérente partout.',
|
||
'- N’utilise JAMAIS d’espaces, accents, chiffres ou symboles dans une abréviation.',
|
||
'',
|
||
'LÉGENDE:',
|
||
'- Renvoie une "legende" : un objet associant CHAQUE abréviation utilisée (entité, attribut, contexte)',
|
||
' à sa signification complète en français. Ex.: {"PER":"Personne","NOM":"Nom de famille","M":"Masculin"}.',
|
||
'- La légende sert à expliquer les placeholders à un autre modèle ; elle ne contient AUCUNE donnée réelle.',
|
||
'',
|
||
'COHÉRENCE (impératif) :',
|
||
'- Chaque clé de "mapping" est EXACTEMENT un placeholder présent dans "texte_anonymise" ; chaque placeholder du texte est une clé de "mapping".',
|
||
'- Les VALEURS de "mapping" sont les données ORIGINALES réelles, JAMAIS un autre placeholder.',
|
||
'- Deux valeurs DIFFÉRENTES ne partagent JAMAIS le même placeholder. Pour deux valeurs du même type chez la même personne (ex. ancien et nouvel IBAN), distingue-les par un CONTEXTE différent : [PER_1.IBAN:Ancien] et [PER_1.IBAN:Nouveau].',
|
||
'- Format OBLIGATOIRE : [TYPE_N] ou [TYPE_N.ATTRIBUT:CONTEXTE] (TYPE en MAJUSCULES, N entier ; ex. [ORG_1.NOM:Entreprise], jamais [ORG:Entreprise]).',
|
||
'- Remplace UNIQUEMENT la valeur elle-même — jamais le libellé voisin (« numéro AVS 756… » → seul « 756… » ; « AVS » reste) NI la ponctuation/séparateurs voisins (virgules, espaces, parenthèses restent HORS des placeholders, ex. « 14 rue X, 1700 Ville » garde la virgule).',
|
||
'',
|
||
'RÈGLES:',
|
||
'1. Coréférence: la MÊME personne garde le MÊME identifiant (PER_1) dans tout le texte.',
|
||
'2. N’anonymise JAMAIS les termes pédagogiques/techniques (langages, concepts, titres de cours, fonctions).',
|
||
'3. Si AUCUNE donnée personnelle: renvoie le texte original, "mapping": {} et "legende": {}.',
|
||
'4. N’anonymise pas un placeholder déjà présent (idempotence).',
|
||
'5. Sortie STRICTEMENT JSON valide:',
|
||
' {"texte_anonymise": "...", "mapping": {"[PER_1.NOM:M]": "..."}, "legende": {"PER": "Personne", "NOM": "Nom de famille", "M": "Masculin"}}.',
|
||
].join('\n');
|
||
|
||
export interface OpenAICompatibleOptions {
|
||
/** Base URL of an OpenAI-compatible API, e.g. `https://api.openai.com/v1`. */
|
||
baseUrl: string;
|
||
/** Bearer API key. */
|
||
apiKey: string;
|
||
/** Model id, e.g. `gpt-4o-mini` or `gemma-3-...`. */
|
||
model: string;
|
||
/** Per-request timeout in milliseconds, per attempt (default 3000). */
|
||
timeoutMs?: number;
|
||
/**
|
||
* Extra retries on TRANSIENT failures only (network error, timeout, HTTP 429/5xx).
|
||
* Default 1 (→ up to 2 attempts). Worst-case latency is `(retries + 1) × timeoutMs`.
|
||
*/
|
||
retries?: number;
|
||
/** Linear backoff between attempts in milliseconds (delay = attempt × this). Default 250. */
|
||
retryDelayMs?: number;
|
||
/**
|
||
* `response_format` to send with the request. **Omitted by default** — the
|
||
* Infomaniak API and several open-model endpoints reject the older
|
||
* `{ type: 'json_object' }`. Pass `{ type: 'json_object' }` for endpoints that
|
||
* require it, or a `{ type: 'json_schema', json_schema: … }` object. Regardless,
|
||
* responses are parsed leniently (markdown fences / surrounding prose tolerated).
|
||
*/
|
||
responseFormat?: Record<string, unknown>;
|
||
/**
|
||
* Include the secret `mapping` (placeholder → real value) in the conversation
|
||
* context during {@link LlmProvider.anonymizeInConversation}. **Default false.**
|
||
* Enable ONLY when this endpoint is a trusted processor — it sends REAL values
|
||
* to the model. (The anonymizer already receives the cleartext being
|
||
* anonymized, so a trusted endpoint like a Swiss/self-hosted deployment sees no
|
||
* more than it already does; the downstream chatbot never receives the mapping.)
|
||
*/
|
||
includeMappingInContext?: boolean;
|
||
/** Override the system prompt (e.g. for another language or regulation). */
|
||
systemPrompt?: string;
|
||
/** Extra instructions appended to the system prompt in batch mode. */
|
||
batchInstructions?: (usedIds: string[]) => string;
|
||
}
|
||
|
||
const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));
|
||
|
||
/** A transport error that the caller may retry (network / timeout / 429 / 5xx). */
|
||
class TransientError extends Error {}
|
||
|
||
/**
|
||
* Parse the model's JSON output leniently: tolerate a ```json fenced block and
|
||
* any prose around the object, keeping the outermost `{ … }`. Models without an
|
||
* enforced `response_format` often wrap their JSON, so this avoids spurious
|
||
* parse failures.
|
||
*/
|
||
function extractJson(content: string): unknown {
|
||
let t = content.trim();
|
||
const fence = t.match(/```(?:json)?\s*([\s\S]*?)```/i);
|
||
if (fence) t = fence[1].trim();
|
||
const i = t.indexOf('{');
|
||
const j = t.lastIndexOf('}');
|
||
if (i >= 0 && j > i) t = t.slice(i, j + 1);
|
||
return JSON.parse(t);
|
||
}
|
||
|
||
/**
|
||
* Coerce the model's `legende` into a string→string map, lenient by design: a
|
||
* missing or malformed legend yields `{}` rather than failing the anonymization
|
||
* (the legend is metadata; the `mapping` is what guarantees no leak).
|
||
*/
|
||
function asLegend(raw: unknown): Record<string, string> {
|
||
if (!raw || typeof raw !== 'object') return {};
|
||
const out: Record<string, string> = {};
|
||
for (const [k, v] of Object.entries(raw as Record<string, unknown>)) {
|
||
if (typeof v === 'string') out[k] = v;
|
||
}
|
||
return out;
|
||
}
|
||
|
||
const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
||
'\n\nMODE LOT (segments) :' +
|
||
'\n- ENTRÉE : un objet JSON {"segments": ["…", "…"]}.' +
|
||
'\n- Anonymise CHAQUE segment ; coréférence GLOBALE entre segments.' +
|
||
(usedIds.length
|
||
? `\n- Identifiants DÉJÀ attribués (réutilise-les pour les mêmes valeurs, n'en duplique AUCUN pour d'autres valeurs) : ${usedIds.join(', ')}.`
|
||
: '') +
|
||
'\n- SORTIE STRICTEMENT JSON : {"segments": ["…anonymisé…"], "mapping": {"[PER_1.NOM:M]": "…"}, "legende": {"PER": "Personne"}}.' +
|
||
'\n- "segments" DOIT avoir la même longueur et le même ordre que l’entrée ; "mapping" ne contient que les NOUVELLES entités ; "legende" décrit toutes les abréviations utilisées.';
|
||
|
||
/** Prior-conversation context block appended to the system prompt for a turn. */
|
||
function conversationContext(
|
||
ctx: {
|
||
history: string[];
|
||
legend: Record<string, string>;
|
||
usedIds: string[];
|
||
mapping?: Record<string, string>;
|
||
},
|
||
includeMapping: boolean,
|
||
): string {
|
||
const lines = ['', 'CONTEXTE DE LA CONVERSATION (déjà pseudonymisé) :'];
|
||
if (ctx.history.length) {
|
||
lines.push('- Tours précédents :', ...ctx.history.map((h, i) => ` [${i + 1}] ${h}`));
|
||
}
|
||
if (Object.keys(ctx.legend).length) {
|
||
lines.push(`- Légende connue : ${JSON.stringify(ctx.legend)}`);
|
||
}
|
||
if (ctx.usedIds.length) {
|
||
lines.push(
|
||
`- Identifiants DÉJÀ attribués (réutilise-les pour les MÊMES entités, n'en réattribue AUCUN à une autre valeur) : ${ctx.usedIds.join(', ')}`,
|
||
);
|
||
}
|
||
if (includeMapping && ctx.mapping && Object.keys(ctx.mapping).length) {
|
||
const pairs = Object.entries(ctx.mapping)
|
||
.map(([ph, v]) => `${ph} = ${v}`)
|
||
.join(' ; ');
|
||
lines.push(
|
||
`- Correspondances connues (réutilise le MÊME placeholder si la valeur réapparaît) : ${pairs}`,
|
||
);
|
||
}
|
||
lines.push(
|
||
'Anonymise le MESSAGE suivant en gardant EXACTEMENT la même convention et les mêmes identifiants pour les entités déjà vues ; "mapping" ne contient que les NOUVELLES entités.',
|
||
);
|
||
return '\n\n' + lines.join('\n');
|
||
}
|
||
|
||
/**
|
||
* Build an {@link LlmProvider} backed by any OpenAI-compatible Chat Completions
|
||
* endpoint (OpenAI, Infomaniak, vLLM, Ollama, …). Uses `temperature: 0` for
|
||
* deterministic output and parses the JSON response leniently. `response_format`
|
||
* is omitted by default (Infomaniak rejects the legacy `json_object`); pass
|
||
* `responseFormat` for endpoints that support/require it. Throws on any failure
|
||
* (so a configured {@link Anonymizer} fallback can take over, or it fails closed).
|
||
*/
|
||
export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProvider {
|
||
const timeout = opts.timeoutMs ?? 3000;
|
||
const retries = Math.max(0, opts.retries ?? 1);
|
||
const retryDelayMs = opts.retryDelayMs ?? 250;
|
||
const responseFormat = opts.responseFormat;
|
||
const includeMappingInContext = opts.includeMappingInContext ?? false;
|
||
const systemPrompt = opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
|
||
const batchInstructions = opts.batchInstructions ?? DEFAULT_BATCH_INSTRUCTIONS;
|
||
|
||
const isConfigured = (): boolean => !!(opts.baseUrl && opts.apiKey && opts.model);
|
||
|
||
/** One HTTP attempt. Throws {@link TransientError} for retryable failures. */
|
||
async function attempt(system: string, user: string): Promise<string> {
|
||
const controller = new AbortController();
|
||
const timer = setTimeout(() => controller.abort(), timeout);
|
||
let res: Response;
|
||
try {
|
||
res = await fetch(`${opts.baseUrl}/chat/completions`, {
|
||
method: 'POST',
|
||
headers: {
|
||
'Content-Type': 'application/json',
|
||
Authorization: `Bearer ${opts.apiKey}`,
|
||
},
|
||
body: JSON.stringify({
|
||
model: opts.model,
|
||
temperature: 0,
|
||
...(responseFormat ? { response_format: responseFormat } : {}),
|
||
messages: [
|
||
{ role: 'system', content: system },
|
||
{ role: 'user', content: user },
|
||
],
|
||
}),
|
||
signal: controller.signal,
|
||
});
|
||
} catch (err) {
|
||
// Network failure or timeout/abort → retryable.
|
||
throw new TransientError(`LLM_FETCH_FAILED: ${(err as Error).message}`);
|
||
} finally {
|
||
clearTimeout(timer);
|
||
}
|
||
if (!res.ok) {
|
||
// 429 + 5xx are transient; other 4xx are not (won't fix on retry).
|
||
if (res.status === 429 || res.status >= 500) throw new TransientError(`LLM_HTTP_${res.status}`);
|
||
throw new Error(`LLM_HTTP_${res.status}`);
|
||
}
|
||
const data = (await res.json()) as { choices?: Array<{ message?: { content?: string } }> };
|
||
return data?.choices?.[0]?.message?.content ?? '';
|
||
}
|
||
|
||
async function chat(system: string, user: string): Promise<string> {
|
||
if (!isConfigured()) throw new Error('LLM_NOT_CONFIGURED');
|
||
let lastErr: unknown;
|
||
for (let i = 0; i <= retries; i++) {
|
||
try {
|
||
return await attempt(system, user);
|
||
} catch (err) {
|
||
lastErr = err;
|
||
if (!(err instanceof TransientError) || i === retries) throw err;
|
||
if (retryDelayMs > 0) await sleep(retryDelayMs * (i + 1));
|
||
}
|
||
}
|
||
throw lastErr; // unreachable, but keeps the type checker happy
|
||
}
|
||
|
||
return {
|
||
isConfigured,
|
||
|
||
async anonymize(text: string): Promise<AnonymizationResult> {
|
||
const content = await chat(systemPrompt, text);
|
||
const parsed = extractJson(content) as {
|
||
texte_anonymise?: string;
|
||
mapping?: Record<string, string>;
|
||
legende?: Record<string, string>;
|
||
};
|
||
if (
|
||
typeof parsed.texte_anonymise !== 'string' ||
|
||
typeof parsed.mapping !== 'object' ||
|
||
!parsed.mapping
|
||
) {
|
||
throw new Error('LLM_BAD_SHAPE');
|
||
}
|
||
return { anon: parsed.texte_anonymise, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||
},
|
||
|
||
async anonymizeInConversation(text, ctx): Promise<AnonymizationResult> {
|
||
const system = systemPrompt + conversationContext(ctx, includeMappingInContext);
|
||
const content = await chat(system, text);
|
||
const parsed = extractJson(content) as {
|
||
texte_anonymise?: string;
|
||
mapping?: Record<string, string>;
|
||
legende?: Record<string, string>;
|
||
};
|
||
if (
|
||
typeof parsed.texte_anonymise !== 'string' ||
|
||
typeof parsed.mapping !== 'object' ||
|
||
!parsed.mapping
|
||
) {
|
||
throw new Error('LLM_BAD_SHAPE');
|
||
}
|
||
return { anon: parsed.texte_anonymise, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||
},
|
||
|
||
async anonymizeBatch(
|
||
texts: string[],
|
||
usedIds: string[],
|
||
): Promise<{ segments: string[]; mapping: Record<string, string>; legend: Record<string, string> }> {
|
||
const system = systemPrompt + batchInstructions(usedIds);
|
||
const content = await chat(system, JSON.stringify({ segments: texts }));
|
||
const parsed = extractJson(content) as {
|
||
segments?: string[];
|
||
mapping?: Record<string, string>;
|
||
legende?: Record<string, string>;
|
||
};
|
||
if (!Array.isArray(parsed.segments) || typeof parsed.mapping !== 'object' || !parsed.mapping) {
|
||
throw new Error('LLM_BATCH_BAD_SHAPE');
|
||
}
|
||
return { segments: parsed.segments, mapping: parsed.mapping, legend: asLegend(parsed.legende) };
|
||
},
|
||
};
|
||
}
|