feat: initial release of @mobiletic/anonymizer
Framework-agnostic PII anonymization extracted from Mobiletic's chatbot. Pluggable LLM detection + configurable regex fallback, deterministic coreference, and streaming-safe de-anonymization. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
24
.github/workflows/ci.yml
vendored
Normal file
24
.github/workflows/ci.yml
vendored
Normal file
@@ -0,0 +1,24 @@
|
|||||||
|
name: CI
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
pull_request:
|
||||||
|
branches: [main]
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
test:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
strategy:
|
||||||
|
matrix:
|
||||||
|
node-version: [18, 20, 22]
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: ${{ matrix.node-version }}
|
||||||
|
cache: npm
|
||||||
|
- run: npm ci
|
||||||
|
- run: npm run typecheck
|
||||||
|
- run: npm test
|
||||||
|
- run: npm run build
|
||||||
28
.github/workflows/release.yml
vendored
Normal file
28
.github/workflows/release.yml
vendored
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
name: Release
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
tags:
|
||||||
|
- 'v*'
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
id-token: write # required for npm provenance
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
publish:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- uses: actions/setup-node@v4
|
||||||
|
with:
|
||||||
|
node-version: 20
|
||||||
|
registry-url: https://registry.npmjs.org
|
||||||
|
cache: npm
|
||||||
|
- run: npm ci
|
||||||
|
- run: npm run typecheck
|
||||||
|
- run: npm test
|
||||||
|
- run: npm run build
|
||||||
|
- run: npm publish --provenance --access public
|
||||||
|
env:
|
||||||
|
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||||
7
.gitignore
vendored
Normal file
7
.gitignore
vendored
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
node_modules/
|
||||||
|
dist/
|
||||||
|
coverage/
|
||||||
|
*.log
|
||||||
|
.DS_Store
|
||||||
|
.env
|
||||||
|
.env.*
|
||||||
17
CHANGELOG.md
Normal file
17
CHANGELOG.md
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
All notable changes to this project are documented here. The format is based on
|
||||||
|
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project adheres to
|
||||||
|
[Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
||||||
|
|
||||||
|
## [0.1.0] - Unreleased
|
||||||
|
|
||||||
|
### Added
|
||||||
|
|
||||||
|
- Initial public release, extracted from Mobiletic's production Swiss-nLPD chatbot.
|
||||||
|
- `Anonymizer` — pre-filter → LLM → regex fallback, bidirectional validation, deterministic coreference,
|
||||||
|
and de-collision across question and retrieved chunks (`anonymize`, `anonymizeChunks`, `deanonymize`).
|
||||||
|
- `makeStreamDeanonymizer` — streaming-safe de-anonymization that never leaks a split placeholder.
|
||||||
|
- `openAICompatibleProvider` — pluggable LLM detection over any OpenAI-compatible Chat Completions API.
|
||||||
|
- `presets.swiss` and `presets.generic` regex pattern sets; fully configurable custom patterns.
|
||||||
|
- Regex-only mode (no LLM provider required).
|
||||||
31
CODE_OF_CONDUCT.md
Normal file
31
CODE_OF_CONDUCT.md
Normal file
@@ -0,0 +1,31 @@
|
|||||||
|
# Contributor Covenant Code of Conduct
|
||||||
|
|
||||||
|
## Our Pledge
|
||||||
|
|
||||||
|
We as members, contributors, and leaders pledge to make participation in our community a harassment-free
|
||||||
|
experience for everyone, regardless of age, body size, visible or invisible disability, ethnicity, sex
|
||||||
|
characteristics, gender identity and expression, level of experience, education, socio-economic status,
|
||||||
|
nationality, personal appearance, race, religion, or sexual identity and orientation.
|
||||||
|
|
||||||
|
We pledge to act and interact in ways that contribute to an open, welcoming, diverse, inclusive, and
|
||||||
|
healthy community.
|
||||||
|
|
||||||
|
## Our Standards
|
||||||
|
|
||||||
|
Examples of behavior that contributes to a positive environment include demonstrating empathy and
|
||||||
|
kindness, being respectful of differing opinions, giving and gracefully accepting constructive feedback,
|
||||||
|
and focusing on what is best for the community.
|
||||||
|
|
||||||
|
Unacceptable behavior includes the use of sexualized language or imagery, trolling or insulting comments,
|
||||||
|
public or private harassment, publishing others' private information without permission, and other conduct
|
||||||
|
which could reasonably be considered inappropriate in a professional setting.
|
||||||
|
|
||||||
|
## Enforcement
|
||||||
|
|
||||||
|
Instances of abusive, harassing, or otherwise unacceptable behavior may be reported to the project team at
|
||||||
|
`conduct@mobiletic.com`. All complaints will be reviewed and investigated promptly and fairly.
|
||||||
|
|
||||||
|
## Attribution
|
||||||
|
|
||||||
|
This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org),
|
||||||
|
version 2.1.
|
||||||
37
CONTRIBUTING.md
Normal file
37
CONTRIBUTING.md
Normal file
@@ -0,0 +1,37 @@
|
|||||||
|
# Contributing to @mobiletic/anonymizer
|
||||||
|
|
||||||
|
Thanks for your interest in improving this project! Contributions of all kinds are welcome — bug reports,
|
||||||
|
new pattern presets, additional LLM providers, docs, and tests.
|
||||||
|
|
||||||
|
## Getting started
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git clone https://github.com/mobiletic/anonymizer.git
|
||||||
|
cd anonymizer
|
||||||
|
npm install
|
||||||
|
npm test # run the test suite (vitest)
|
||||||
|
npm run typecheck # tsc --noEmit
|
||||||
|
npm run build # tsup → dist/ (ESM + CJS + types)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Guidelines
|
||||||
|
|
||||||
|
- **Tests first.** Every behavior change needs a test. The anonymization core is privacy-critical — we
|
||||||
|
keep coverage tight, especially around coreference, de-collision, and streaming.
|
||||||
|
- **No runtime dependencies.** The core must stay dependency-free. New providers should rely only on
|
||||||
|
platform APIs (e.g. `fetch`).
|
||||||
|
- **Keep it framework-agnostic.** No framework-specific code (NestJS, Express, React, …) in the package.
|
||||||
|
- **Patterns use the global flag.** Any `PatternDef.re` must be a global (`/…/g`) regex.
|
||||||
|
- **Conventional commits** are appreciated (`feat:`, `fix:`, `docs:`, `test:`, `chore:`).
|
||||||
|
|
||||||
|
## Reporting security issues
|
||||||
|
|
||||||
|
Please do **not** open public issues for vulnerabilities (e.g. a PII-leak path). Email
|
||||||
|
`security@mobiletic.com` instead.
|
||||||
|
|
||||||
|
## Pull requests
|
||||||
|
|
||||||
|
1. Fork and branch from `main`.
|
||||||
|
2. Add tests and update docs.
|
||||||
|
3. Ensure `npm test`, `npm run typecheck`, and `npm run build` pass.
|
||||||
|
4. Open the PR with a clear description of the change and its motivation.
|
||||||
21
LICENSE
Normal file
21
LICENSE
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
MIT License
|
||||||
|
|
||||||
|
Copyright (c) 2026 Mobiletic
|
||||||
|
|
||||||
|
Permission is hereby granted, free of charge, to any person obtaining a copy
|
||||||
|
of this software and associated documentation files (the "Software"), to deal
|
||||||
|
in the Software without restriction, including without limitation the rights
|
||||||
|
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||||
|
copies of the Software, and to permit persons to whom the Software is
|
||||||
|
furnished to do so, subject to the following conditions:
|
||||||
|
|
||||||
|
The above copyright notice and this permission notice shall be included in all
|
||||||
|
copies or substantial portions of the Software.
|
||||||
|
|
||||||
|
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||||
|
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
||||||
|
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||||
|
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||||
|
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||||
|
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
||||||
|
SOFTWARE.
|
||||||
146
README.md
Normal file
146
README.md
Normal file
@@ -0,0 +1,146 @@
|
|||||||
|
# @mobiletic/anonymizer
|
||||||
|
|
||||||
|
Framework-agnostic **PII anonymization & pseudonymization** for TypeScript/JavaScript.
|
||||||
|
|
||||||
|
It replaces personal data in text with stable placeholders before the text leaves your trust boundary
|
||||||
|
(e.g. before sending it to a third-party LLM, log sink, or analytics pipeline), and restores the real
|
||||||
|
values afterwards — including across **streamed** tokens.
|
||||||
|
|
||||||
|
- 🔌 **Pluggable LLM detection** — catch free-form PII (names, addresses) via any OpenAI-compatible
|
||||||
|
endpoint, or your own provider. No LLM? It degrades gracefully to regex.
|
||||||
|
- 🧩 **Deterministic regex fallback** — structured identifiers (email, phone, IBAN, …) via configurable
|
||||||
|
pattern presets (`swiss`, `generic`) or your own.
|
||||||
|
- 🔁 **Deterministic coreference** — the same person keeps the same id (`[PER_1]`) across a question and
|
||||||
|
every retrieved chunk, with automatic de-collision.
|
||||||
|
- 🌊 **Streaming-safe** — a placeholder split across two stream chunks (`[PER_` + `1.NOM:M]`) is never
|
||||||
|
leaked partially.
|
||||||
|
- 🪶 **Zero runtime dependencies**, ESM + CJS, fully typed.
|
||||||
|
|
||||||
|
> Built by [Mobiletic](https://mobiletic.com) and extracted from a production Swiss-nLPD chatbot.
|
||||||
|
|
||||||
|
## Install
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install @mobiletic/anonymizer
|
||||||
|
```
|
||||||
|
|
||||||
|
Requires Node ≥ 18 (uses native `fetch`).
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
### Regex-only (no LLM, fully deterministic)
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { Anonymizer, presets } from '@mobiletic/anonymizer';
|
||||||
|
|
||||||
|
const anonymizer = new Anonymizer({ patterns: presets.swiss });
|
||||||
|
|
||||||
|
const { anon, mapping } = await anonymizer.anonymize('Écris à jean@exemple.ch');
|
||||||
|
// anon -> "Écris à [EMAIL_1]"
|
||||||
|
// mapping -> { "[EMAIL_1]": "jean@exemple.ch" }
|
||||||
|
|
||||||
|
anonymizer.deanonymize(anon, mapping); // -> "Écris à jean@exemple.ch"
|
||||||
|
```
|
||||||
|
|
||||||
|
### With an LLM (also catches names, addresses…)
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { Anonymizer, openAICompatibleProvider, presets } from '@mobiletic/anonymizer';
|
||||||
|
|
||||||
|
const anonymizer = new Anonymizer({
|
||||||
|
llm: openAICompatibleProvider({
|
||||||
|
baseUrl: process.env.LLM_BASE_URL!, // OpenAI, Infomaniak, vLLM, Ollama, …
|
||||||
|
apiKey: process.env.LLM_API_KEY!,
|
||||||
|
model: process.env.LLM_MODEL!,
|
||||||
|
timeoutMs: 3000,
|
||||||
|
}),
|
||||||
|
patterns: presets.swiss, // regex fallback if the LLM is down/misbehaves
|
||||||
|
});
|
||||||
|
|
||||||
|
const { anon, mapping } = await anonymizer.anonymize('Le dossier de Alain Jaccard est complet.');
|
||||||
|
// anon -> "Le dossier de [PER_1.NOM:M] est complet."
|
||||||
|
```
|
||||||
|
|
||||||
|
If the LLM call fails, times out, or returns an invalid shape, the anonymizer automatically falls back to
|
||||||
|
the regex engine — it never throws on a provider failure.
|
||||||
|
|
||||||
|
### Streaming de-anonymization
|
||||||
|
|
||||||
|
When you stream an LLM answer back to a user, restore real values without ever emitting a half-written
|
||||||
|
placeholder:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const stream = anonymizer.makeStreamDeanonymizer(mapping);
|
||||||
|
for await (const token of llmTokens) process.stdout.write(stream.push(token));
|
||||||
|
process.stdout.write(stream.flush());
|
||||||
|
```
|
||||||
|
|
||||||
|
### Anonymizing retrieved chunks consistently (RAG)
|
||||||
|
|
||||||
|
`anonymizeChunks(chunks, seed)` reuses the question's mapping (`seed`) so the same person gets the same id
|
||||||
|
across the question and every chunk, batches the LLM call, and de-collides genuinely new entities:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
const q = await anonymizer.anonymize(question);
|
||||||
|
const { anon, mapping } = await anonymizer.anonymizeChunks(retrievedChunks, q.mapping);
|
||||||
|
// `mapping` is the full question ∪ chunks table; pass it to deanonymize()/makeStreamDeanonymizer().
|
||||||
|
```
|
||||||
|
|
||||||
|
## Configuration
|
||||||
|
|
||||||
|
```ts
|
||||||
|
new Anonymizer({
|
||||||
|
llm?, // LlmProvider — omit for regex-only mode
|
||||||
|
patterns?, // PatternDef[] — defaults to presets.swiss
|
||||||
|
nameHint?, // RegExp flagging likely names so the LLM is consulted (has a default)
|
||||||
|
logger?, // { warn(msg) } — receives fallback warnings; defaults to no-op
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
### Presets & custom patterns
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import { presets } from '@mobiletic/anonymizer';
|
||||||
|
|
||||||
|
presets.swiss; // AVS, IBAN CH, EMAIL, Swiss phone, DATE
|
||||||
|
presets.generic; // EMAIL, IBAN, credit card, IPv4, phone, DATE
|
||||||
|
|
||||||
|
// Compose / extend:
|
||||||
|
const patterns = [
|
||||||
|
...presets.generic,
|
||||||
|
{ tag: 'TICKET', re: /\bJIRA-\d+\b/g }, // patterns must use the global flag
|
||||||
|
];
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom LLM provider
|
||||||
|
|
||||||
|
Implement `LlmProvider` to use any backend (Anthropic, a local model, a rules engine…):
|
||||||
|
|
||||||
|
```ts
|
||||||
|
import type { LlmProvider } from '@mobiletic/anonymizer';
|
||||||
|
|
||||||
|
const myProvider: LlmProvider = {
|
||||||
|
isConfigured: () => true,
|
||||||
|
async anonymize(text) { /* return { anon, mapping } */ },
|
||||||
|
async anonymizeBatch(texts, usedIds) { /* return { segments, mapping } */ },
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
## Placeholder format
|
||||||
|
|
||||||
|
```
|
||||||
|
[PER_1.NOM:M] entity PER #1, attribute NOM, context M (rich, from the LLM)
|
||||||
|
[EMAIL_1] structured id from the regex fallback
|
||||||
|
```
|
||||||
|
|
||||||
|
`PLACEHOLDER_RE` is exported if you need to scan text for placeholders.
|
||||||
|
|
||||||
|
## Compliance note
|
||||||
|
|
||||||
|
This library is a **best-effort** pseudonymization aid, not a guarantee of legal compliance. LLM and regex
|
||||||
|
detection can miss or mis-classify data. Validate against your own requirements (nLPD, GDPR, HIPAA, …)
|
||||||
|
before relying on it for regulated data.
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
[MIT](./LICENSE) © Mobiletic
|
||||||
2791
package-lock.json
generated
Normal file
2791
package-lock.json
generated
Normal file
File diff suppressed because it is too large
Load Diff
62
package.json
Normal file
62
package.json
Normal file
@@ -0,0 +1,62 @@
|
|||||||
|
{
|
||||||
|
"name": "@mobiletic/anonymizer",
|
||||||
|
"version": "0.1.0",
|
||||||
|
"description": "Framework-agnostic PII anonymization & pseudonymization: pluggable LLM detection for free-form PII (names, addresses) with a deterministic regex fallback, deterministic coreference, and streaming-safe de-anonymization.",
|
||||||
|
"license": "MIT",
|
||||||
|
"author": "Mobiletic",
|
||||||
|
"homepage": "https://github.com/mobiletic/anonymizer#readme",
|
||||||
|
"repository": {
|
||||||
|
"type": "git",
|
||||||
|
"url": "git+https://github.com/mobiletic/anonymizer.git"
|
||||||
|
},
|
||||||
|
"bugs": {
|
||||||
|
"url": "https://github.com/mobiletic/anonymizer/issues"
|
||||||
|
},
|
||||||
|
"keywords": [
|
||||||
|
"anonymization",
|
||||||
|
"pseudonymization",
|
||||||
|
"pii",
|
||||||
|
"redaction",
|
||||||
|
"privacy",
|
||||||
|
"gdpr",
|
||||||
|
"nlpd",
|
||||||
|
"llm",
|
||||||
|
"data-protection"
|
||||||
|
],
|
||||||
|
"type": "module",
|
||||||
|
"main": "./dist/index.cjs",
|
||||||
|
"module": "./dist/index.js",
|
||||||
|
"types": "./dist/index.d.ts",
|
||||||
|
"exports": {
|
||||||
|
".": {
|
||||||
|
"types": "./dist/index.d.ts",
|
||||||
|
"import": "./dist/index.js",
|
||||||
|
"require": "./dist/index.cjs"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"files": [
|
||||||
|
"dist",
|
||||||
|
"README.md",
|
||||||
|
"LICENSE"
|
||||||
|
],
|
||||||
|
"engines": {
|
||||||
|
"node": ">=18"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"build": "tsup",
|
||||||
|
"test": "vitest run",
|
||||||
|
"test:watch": "vitest",
|
||||||
|
"typecheck": "tsc --noEmit",
|
||||||
|
"prepublishOnly": "npm run build"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"tsup": "^8.0.0",
|
||||||
|
"typescript": "^5.4.0",
|
||||||
|
"vitest": "^1.6.0"
|
||||||
|
},
|
||||||
|
"publishConfig": {
|
||||||
|
"access": "public",
|
||||||
|
"provenance": true
|
||||||
|
},
|
||||||
|
"sideEffects": false
|
||||||
|
}
|
||||||
244
src/anonymizer.ts
Normal file
244
src/anonymizer.ts
Normal file
@@ -0,0 +1,244 @@
|
|||||||
|
import { RegexFallback } from './fallback.js';
|
||||||
|
import { presets } from './presets.js';
|
||||||
|
import {
|
||||||
|
type AnonymizationResult,
|
||||||
|
type AnonymizerConfig,
|
||||||
|
type LlmProvider,
|
||||||
|
type Logger,
|
||||||
|
type StreamDeanonymizer,
|
||||||
|
PLACEHOLDER_RE,
|
||||||
|
} from './types.js';
|
||||||
|
|
||||||
|
const NOOP_LOGGER: Logger = { warn: () => {} };
|
||||||
|
|
||||||
|
// Likely proper-name heuristic (two capitalized words), in addition to the structured fallback.
|
||||||
|
const DEFAULT_NAME_HINT = /\b[A-ZÀ-Ý][a-zà-ÿ]+\s+[A-ZÀ-Ý][a-zà-ÿ]+\b/;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Framework-agnostic anonymization engine.
|
||||||
|
*
|
||||||
|
* `anonymize()`: pre-filter (skips the LLM when there is no PII) → LLM →
|
||||||
|
* regex fallback on failure/timeout → bidirectional validation.
|
||||||
|
* `deanonymize()` restores the real values. `makeStreamDeanonymizer()` handles
|
||||||
|
* the critical case of a placeholder fragmented across several streamed tokens.
|
||||||
|
*
|
||||||
|
* Pass an {@link LlmProvider} to detect free-form PII (names, addresses);
|
||||||
|
* omit it for deterministic regex-only mode.
|
||||||
|
*/
|
||||||
|
export class Anonymizer {
|
||||||
|
private readonly llm?: LlmProvider;
|
||||||
|
private readonly fallback: RegexFallback;
|
||||||
|
private readonly nameHint: RegExp;
|
||||||
|
private readonly logger: Logger;
|
||||||
|
|
||||||
|
constructor(config: AnonymizerConfig = {}) {
|
||||||
|
this.llm = config.llm;
|
||||||
|
this.fallback = new RegexFallback(config.patterns ?? presets.swiss);
|
||||||
|
this.nameHint = config.nameHint ?? DEFAULT_NAME_HINT;
|
||||||
|
this.logger = config.logger ?? NOOP_LOGGER;
|
||||||
|
}
|
||||||
|
|
||||||
|
private hasLlm(): boolean {
|
||||||
|
return !!this.llm && this.llm.isConfigured();
|
||||||
|
}
|
||||||
|
|
||||||
|
private likelyHasPii(text: string): boolean {
|
||||||
|
return this.fallback.hasPii(text) || this.nameHint.test(text);
|
||||||
|
}
|
||||||
|
|
||||||
|
async anonymize(text: string): Promise<AnonymizationResult> {
|
||||||
|
// Pre-filter: most text has no PII → avoid the LLM round-trip (latency + cost)
|
||||||
|
// and return the text unchanged.
|
||||||
|
if (!this.likelyHasPii(text)) return { anon: text, mapping: {} };
|
||||||
|
|
||||||
|
if (this.hasLlm()) {
|
||||||
|
try {
|
||||||
|
const result = await this.llm!.anonymize(text);
|
||||||
|
this.validate(result); // throws if inconsistent → fall back
|
||||||
|
return result;
|
||||||
|
} catch (err) {
|
||||||
|
this.logger.warn(`LLM unavailable/invalid → regex fallback: ${(err as Error).message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return this.fallback.anonymize(text);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Anonymize retrieved chunks BEFORE sending them on, CONSISTENTLY with the
|
||||||
|
* question's mapping (`seed`). Returns the anonymized chunks + the COMPLETE
|
||||||
|
* mapping (question ∪ chunks). Strategy:
|
||||||
|
* 1. Deterministic reuse: replace any already-known value (from the question)
|
||||||
|
* by its placeholder → question↔chunk coreference with no LLM call.
|
||||||
|
* 2. Pre-filter: if no residual PII remains → return as-is (most corpus chunks
|
||||||
|
* have no PII → LLM cost avoided).
|
||||||
|
* 3. Otherwise: ONE batched LLM call, then de-collision merge (renumber the
|
||||||
|
* genuinely new entities). Failure/timeout → regex fallback per chunk.
|
||||||
|
*/
|
||||||
|
async anonymizeChunks(
|
||||||
|
chunks: string[],
|
||||||
|
seed: Record<string, string>,
|
||||||
|
): Promise<{ anon: string[]; mapping: Record<string, string> }> {
|
||||||
|
if (!chunks.length) return { anon: [], mapping: { ...seed } };
|
||||||
|
|
||||||
|
const seeded = chunks.map((c) => this.applyKnown(c, seed));
|
||||||
|
if (!seeded.some((c) => this.likelyHasPii(c))) return { anon: seeded, mapping: { ...seed } };
|
||||||
|
|
||||||
|
if (this.hasLlm()) {
|
||||||
|
try {
|
||||||
|
const { segments, mapping: gMap } = await this.llm!.anonymizeBatch(seeded, Object.keys(seed));
|
||||||
|
if (segments.length !== seeded.length) throw new Error('SEGMENT_COUNT_MISMATCH');
|
||||||
|
const { mapping, rename } = this.mergeMappings(seed, gMap);
|
||||||
|
const anon = segments.map((s) => this.applyRename(s, rename));
|
||||||
|
// Anti-leak: every placeholder present in a chunk must be mapped.
|
||||||
|
for (const s of anon) {
|
||||||
|
for (const ph of s.match(PLACEHOLDER_RE) ?? []) {
|
||||||
|
if (!(ph in mapping)) throw new Error(`unmapped placeholder ${ph}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { anon, mapping };
|
||||||
|
} catch (err) {
|
||||||
|
this.logger.warn(`Chunk anonymization → regex fallback: ${(err as Error).message}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return this.fallbackChunks(seeded, seed);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Replace known values (seed) by their placeholder, longest values first. */
|
||||||
|
private applyKnown(text: string, seed: Record<string, string>): string {
|
||||||
|
let out = text;
|
||||||
|
for (const [ph, val] of Object.entries(seed)
|
||||||
|
.filter(([, v]) => v)
|
||||||
|
.sort((a, b) => b[1].length - a[1].length)) {
|
||||||
|
out = out.split(val).join(ph);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
private applyRename(text: string, rename: Record<string, string>): string {
|
||||||
|
let out = text;
|
||||||
|
for (const [from, to] of Object.entries(rename)) out = out.split(from).join(to);
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** `[PER_1.NOM:M]` → { type:'PER', index:1, suffix:'.NOM:M' } ; `[EMAIL_2]` → suffix ''. */
|
||||||
|
private parsePlaceholder(ph: string): { type: string; index: number; suffix: string } | null {
|
||||||
|
const m = /^\[([A-Z]+)_(\d+)(\..*)?\]$/.exec(ph);
|
||||||
|
return m ? { type: m[1], index: Number(m[2]), suffix: m[3] ?? '' } : null;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Merge the chunks' LLM mapping into the question's (seed).
|
||||||
|
* - value already known → reuse the question's placeholder (rename).
|
||||||
|
* - new value without collision → added as-is.
|
||||||
|
* - collision (same placeholder, different value) → renumbered (next index).
|
||||||
|
*/
|
||||||
|
private mergeMappings(
|
||||||
|
seed: Record<string, string>,
|
||||||
|
gMap: Record<string, string>,
|
||||||
|
): { mapping: Record<string, string>; rename: Record<string, string> } {
|
||||||
|
const mapping: Record<string, string> = { ...seed };
|
||||||
|
const valueToPh = new Map<string, string>();
|
||||||
|
const maxIdx = new Map<string, number>();
|
||||||
|
for (const [ph, val] of Object.entries(seed)) {
|
||||||
|
valueToPh.set(val, ph);
|
||||||
|
const p = this.parsePlaceholder(ph);
|
||||||
|
if (p) maxIdx.set(p.type, Math.max(maxIdx.get(p.type) ?? 0, p.index));
|
||||||
|
}
|
||||||
|
|
||||||
|
const rename: Record<string, string> = {};
|
||||||
|
for (const [gph, val] of Object.entries(gMap)) {
|
||||||
|
const known = valueToPh.get(val);
|
||||||
|
if (known) {
|
||||||
|
if (known !== gph) rename[gph] = known; // same entity as the question
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (!(gph in mapping)) {
|
||||||
|
mapping[gph] = val;
|
||||||
|
valueToPh.set(val, gph);
|
||||||
|
const p = this.parsePlaceholder(gph);
|
||||||
|
if (p) maxIdx.set(p.type, Math.max(maxIdx.get(p.type) ?? 0, p.index));
|
||||||
|
} else {
|
||||||
|
// collision: placeholder reused for ANOTHER value → renumber.
|
||||||
|
const p = this.parsePlaceholder(gph);
|
||||||
|
const type = p?.type ?? 'PER';
|
||||||
|
const next = (maxIdx.get(type) ?? 0) + 1;
|
||||||
|
maxIdx.set(type, next);
|
||||||
|
const newPh = `[${type}_${next}${p?.suffix ?? ''}]`;
|
||||||
|
rename[gph] = newPh;
|
||||||
|
mapping[newPh] = val;
|
||||||
|
valueToPh.set(val, newPh);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return { mapping, rename };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Fallback (no LLM): regex per chunk, de-collision merged into the seed. */
|
||||||
|
private fallbackChunks(
|
||||||
|
seeded: string[],
|
||||||
|
seed: Record<string, string>,
|
||||||
|
): { anon: string[]; mapping: Record<string, string> } {
|
||||||
|
let mapping: Record<string, string> = { ...seed };
|
||||||
|
const anon: string[] = [];
|
||||||
|
for (const c of seeded) {
|
||||||
|
const r = this.fallback.anonymize(c);
|
||||||
|
const { mapping: merged, rename } = this.mergeMappings(mapping, r.mapping);
|
||||||
|
mapping = merged;
|
||||||
|
anon.push(this.applyRename(r.anon, rename));
|
||||||
|
}
|
||||||
|
return { anon, mapping };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Every placeholder in the text must be in the mapping AND vice-versa (anti-leak). */
|
||||||
|
private validate(result: AnonymizationResult): void {
|
||||||
|
const inText = new Set(result.anon.match(PLACEHOLDER_RE) ?? []);
|
||||||
|
for (const ph of inText) {
|
||||||
|
if (!(ph in result.mapping)) throw new Error(`placeholder ${ph} without mapping`);
|
||||||
|
}
|
||||||
|
for (const ph of Object.keys(result.mapping)) {
|
||||||
|
if (!inText.has(ph)) throw new Error(`orphan mapping ${ph}`);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Restore the real values (replace every known placeholder). */
|
||||||
|
deanonymize(text: string, mapping: Record<string, string>): string {
|
||||||
|
if (!text) return text;
|
||||||
|
let out = text;
|
||||||
|
for (const [ph, value] of Object.entries(mapping)) {
|
||||||
|
out = out.split(ph).join(value);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Streaming de-anonymizer. Buffers any unterminated `[...]` fragment until it
|
||||||
|
* completes, so a partial placeholder is never emitted and never leaks.
|
||||||
|
*/
|
||||||
|
makeStreamDeanonymizer(mapping: Record<string, string>): StreamDeanonymizer {
|
||||||
|
let buf = '';
|
||||||
|
const deanon = (s: string) => this.deanonymize(s, mapping);
|
||||||
|
return {
|
||||||
|
push: (chunk: string): string => {
|
||||||
|
buf += chunk;
|
||||||
|
const lastOpen = buf.lastIndexOf('[');
|
||||||
|
let emit: string;
|
||||||
|
if (lastOpen === -1) {
|
||||||
|
emit = buf;
|
||||||
|
buf = '';
|
||||||
|
} else if (buf.indexOf(']', lastOpen) === -1) {
|
||||||
|
// '[' opened without ']' → placeholder possibly cut: hold from there.
|
||||||
|
emit = buf.slice(0, lastOpen);
|
||||||
|
buf = buf.slice(lastOpen);
|
||||||
|
} else {
|
||||||
|
emit = buf;
|
||||||
|
buf = '';
|
||||||
|
}
|
||||||
|
return deanon(emit);
|
||||||
|
},
|
||||||
|
flush: (): string => {
|
||||||
|
const out = deanon(buf);
|
||||||
|
buf = '';
|
||||||
|
return out;
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
|
}
|
||||||
40
src/fallback.ts
Normal file
40
src/fallback.ts
Normal file
@@ -0,0 +1,40 @@
|
|||||||
|
import type { AnonymizationResult, PatternDef } from './types.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Regex-based anonymizer for STRUCTURED identifiers (email, phone, IBAN, …).
|
||||||
|
* It is the deterministic fallback used when no LLM provider is available or a
|
||||||
|
* provider call fails. It does NOT detect proper names — that is the LLM's job.
|
||||||
|
*
|
||||||
|
* Patterns are injected (see {@link presets}) so the same engine serves any
|
||||||
|
* locale or regulation.
|
||||||
|
*/
|
||||||
|
export class RegexFallback {
|
||||||
|
constructor(private readonly patterns: PatternDef[]) {}
|
||||||
|
|
||||||
|
anonymize(text: string): AnonymizationResult {
|
||||||
|
const mapping: Record<string, string> = {};
|
||||||
|
const counters: Record<string, number> = {};
|
||||||
|
let anon = text;
|
||||||
|
|
||||||
|
for (const { tag, re } of this.patterns) {
|
||||||
|
anon = anon.replace(re, (match) => {
|
||||||
|
// Reuse the same placeholder for an identical value (consistency).
|
||||||
|
const existing = Object.entries(mapping).find(([, v]) => v === match);
|
||||||
|
if (existing) return existing[0];
|
||||||
|
counters[tag] = (counters[tag] ?? 0) + 1;
|
||||||
|
const ph = `[${tag}_${counters[tag]}]`;
|
||||||
|
mapping[ph] = match;
|
||||||
|
return ph;
|
||||||
|
});
|
||||||
|
}
|
||||||
|
return { anon, mapping };
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Fast pre-check: is there any structured identifier worth anonymizing? */
|
||||||
|
hasPii(text: string): boolean {
|
||||||
|
return this.patterns.some(({ re }) => {
|
||||||
|
re.lastIndex = 0;
|
||||||
|
return re.test(text);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
17
src/index.ts
Normal file
17
src/index.ts
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
export { Anonymizer } from './anonymizer.js';
|
||||||
|
export { RegexFallback } from './fallback.js';
|
||||||
|
export { presets } from './presets.js';
|
||||||
|
export {
|
||||||
|
openAICompatibleProvider,
|
||||||
|
DEFAULT_SYSTEM_PROMPT,
|
||||||
|
type OpenAICompatibleOptions,
|
||||||
|
} from './providers/openai-compatible.js';
|
||||||
|
export {
|
||||||
|
PLACEHOLDER_RE,
|
||||||
|
type AnonymizationResult,
|
||||||
|
type AnonymizerConfig,
|
||||||
|
type LlmProvider,
|
||||||
|
type PatternDef,
|
||||||
|
type Logger,
|
||||||
|
type StreamDeanonymizer,
|
||||||
|
} from './types.js';
|
||||||
29
src/presets.ts
Normal file
29
src/presets.ts
Normal file
@@ -0,0 +1,29 @@
|
|||||||
|
import type { PatternDef } from './types.js';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Swiss preset — the structured identifiers most relevant to the Swiss nLPD.
|
||||||
|
* Order matters: specific patterns (AVS, IBAN) come before generic ones.
|
||||||
|
*/
|
||||||
|
const swiss: PatternDef[] = [
|
||||||
|
{ tag: 'AVS', re: /\b756\.\d{4}\.\d{4}\.\d{2}\b/g }, // Swiss social security (AVS/AHV)
|
||||||
|
{ tag: 'IBAN', re: /\bCH\d{2}[0-9A-Z]{17}\b/gi }, // Swiss IBAN
|
||||||
|
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g },
|
||||||
|
{ tag: 'TEL', re: /(?:\+41|0)(?:[\s.-]?\d){9}\b/g }, // Swiss phone number
|
||||||
|
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
||||||
|
];
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Generic preset — locale-agnostic identifiers useful as a starting point
|
||||||
|
* anywhere. Extend or compose with your own {@link PatternDef}s as needed.
|
||||||
|
*/
|
||||||
|
const generic: PatternDef[] = [
|
||||||
|
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g },
|
||||||
|
{ tag: 'IBAN', re: /\b[A-Z]{2}\d{2}[0-9A-Z]{11,30}\b/g },
|
||||||
|
{ tag: 'CREDIT_CARD', re: /\b(?:\d[ -]?){13,16}\b/g },
|
||||||
|
{ tag: 'IPV4', re: /\b(?:\d{1,3}\.){3}\d{1,3}\b/g },
|
||||||
|
{ tag: 'TEL', re: /\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]?){2,4}\d{2,4}\b/g },
|
||||||
|
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
||||||
|
];
|
||||||
|
|
||||||
|
/** Built-in pattern sets for the regex fallback. */
|
||||||
|
export const presets = { swiss, generic };
|
||||||
117
src/providers/openai-compatible.ts
Normal file
117
src/providers/openai-compatible.ts
Normal file
@@ -0,0 +1,117 @@
|
|||||||
|
import type { AnonymizationResult, LlmProvider } from '../types.js';
|
||||||
|
|
||||||
|
/** Default nLPD pseudonymization prompt (French). Override for other locales/regulations. */
|
||||||
|
export const DEFAULT_SYSTEM_PROMPT = [
|
||||||
|
'Tu es un moteur de pseudonymisation conforme à la nLPD suisse.',
|
||||||
|
'Identifie UNIQUEMENT les données personnelles (identifiants directs et indirects)',
|
||||||
|
'et remplace-les par des placeholders. Ne touche à RIEN d’autre.',
|
||||||
|
'',
|
||||||
|
'FORMAT: [ENTITE_ID.ATTRIBUT:CONTEXTE]',
|
||||||
|
'Entités: PER (personne), ORG (organisation), LOC (lieu autonome).',
|
||||||
|
'Attributs PER: NOM, PRENOM, DATE_NAISSANCE, AGE, ADRESSE, EMAIL, TELEPHONE, AVS, IBAN, NSS.',
|
||||||
|
'Contexte = indice non-identifiant utile au raisonnement:',
|
||||||
|
' NOM:M|F|U · DATE_NAISSANCE:<année> · AGE:Mineur|Adulte',
|
||||||
|
' ADRESSE:Lieu|Rue|Ville|NPA|Pays · ORG:Entreprise|Ecole',
|
||||||
|
'',
|
||||||
|
'RÈGLES:',
|
||||||
|
'1. Coréférence: la MÊME personne garde le MÊME identifiant (PER_1) dans tout le texte.',
|
||||||
|
'2. N’anonymise JAMAIS les termes pédagogiques/techniques (langages, concepts, titres de cours, fonctions).',
|
||||||
|
'3. Si AUCUNE donnée personnelle: renvoie le texte original et "mapping": {}.',
|
||||||
|
'4. N’anonymise pas un placeholder déjà présent (idempotence).',
|
||||||
|
'5. Sortie STRICTEMENT JSON valide: {"texte_anonymise": "...", "mapping": {"[PER_1.NOM:M]": "..."}}.',
|
||||||
|
].join('\n');
|
||||||
|
|
||||||
|
export interface OpenAICompatibleOptions {
|
||||||
|
/** Base URL of an OpenAI-compatible API, e.g. `https://api.openai.com/v1`. */
|
||||||
|
baseUrl: string;
|
||||||
|
/** Bearer API key. */
|
||||||
|
apiKey: string;
|
||||||
|
/** Model id, e.g. `gpt-4o-mini` or `gemma-3-...`. */
|
||||||
|
model: string;
|
||||||
|
/** Per-request timeout in milliseconds (default 3000). */
|
||||||
|
timeoutMs?: number;
|
||||||
|
/** Override the system prompt (e.g. for another language or regulation). */
|
||||||
|
systemPrompt?: string;
|
||||||
|
/** Extra instructions appended to the system prompt in batch mode. */
|
||||||
|
batchInstructions?: (usedIds: string[]) => string;
|
||||||
|
}
|
||||||
|
|
||||||
|
const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
||||||
|
'\n\nMODE LOT (segments) :' +
|
||||||
|
'\n- ENTRÉE : un objet JSON {"segments": ["…", "…"]}.' +
|
||||||
|
'\n- Anonymise CHAQUE segment ; coréférence GLOBALE entre segments.' +
|
||||||
|
(usedIds.length
|
||||||
|
? `\n- Identifiants DÉJÀ attribués (réutilise-les pour les mêmes valeurs, n'en duplique AUCUN pour d'autres valeurs) : ${usedIds.join(', ')}.`
|
||||||
|
: '') +
|
||||||
|
'\n- SORTIE STRICTEMENT JSON : {"segments": ["…anonymisé…"], "mapping": {"[PER_1.NOM:M]": "…"}}.' +
|
||||||
|
'\n- "segments" DOIT avoir la même longueur et le même ordre que l’entrée ; "mapping" ne contient que les NOUVELLES entités.';
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Build an {@link LlmProvider} backed by any OpenAI-compatible Chat Completions
|
||||||
|
* endpoint (OpenAI, Infomaniak, vLLM, Ollama, …). Uses `temperature: 0` and
|
||||||
|
* `response_format: json_object` for deterministic, parseable output, and throws
|
||||||
|
* on any failure so the {@link Anonymizer} falls back to its regex engine.
|
||||||
|
*/
|
||||||
|
export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProvider {
|
||||||
|
const timeout = opts.timeoutMs ?? 3000;
|
||||||
|
const systemPrompt = opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
|
||||||
|
const batchInstructions = opts.batchInstructions ?? DEFAULT_BATCH_INSTRUCTIONS;
|
||||||
|
|
||||||
|
const isConfigured = (): boolean => !!(opts.baseUrl && opts.apiKey && opts.model);
|
||||||
|
|
||||||
|
async function chat(system: string, user: string): Promise<string> {
|
||||||
|
if (!isConfigured()) throw new Error('LLM_NOT_CONFIGURED');
|
||||||
|
const controller = new AbortController();
|
||||||
|
const timer = setTimeout(() => controller.abort(), timeout);
|
||||||
|
try {
|
||||||
|
const res = await fetch(`${opts.baseUrl}/chat/completions`, {
|
||||||
|
method: 'POST',
|
||||||
|
headers: {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
Authorization: `Bearer ${opts.apiKey}`,
|
||||||
|
},
|
||||||
|
body: JSON.stringify({
|
||||||
|
model: opts.model,
|
||||||
|
temperature: 0,
|
||||||
|
response_format: { type: 'json_object' },
|
||||||
|
messages: [
|
||||||
|
{ role: 'system', content: system },
|
||||||
|
{ role: 'user', content: user },
|
||||||
|
],
|
||||||
|
}),
|
||||||
|
signal: controller.signal,
|
||||||
|
});
|
||||||
|
if (!res.ok) throw new Error(`LLM_HTTP_${res.status}`);
|
||||||
|
const data = (await res.json()) as { choices?: Array<{ message?: { content?: string } }> };
|
||||||
|
return data?.choices?.[0]?.message?.content ?? '';
|
||||||
|
} finally {
|
||||||
|
clearTimeout(timer);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return {
|
||||||
|
isConfigured,
|
||||||
|
|
||||||
|
async anonymize(text: string): Promise<AnonymizationResult> {
|
||||||
|
const content = await chat(systemPrompt, text);
|
||||||
|
const parsed = JSON.parse(content) as { texte_anonymise?: string; mapping?: Record<string, string> };
|
||||||
|
if (typeof parsed.texte_anonymise !== 'string' || typeof parsed.mapping !== 'object' || !parsed.mapping) {
|
||||||
|
throw new Error('LLM_BAD_SHAPE');
|
||||||
|
}
|
||||||
|
return { anon: parsed.texte_anonymise, mapping: parsed.mapping };
|
||||||
|
},
|
||||||
|
|
||||||
|
async anonymizeBatch(
|
||||||
|
texts: string[],
|
||||||
|
usedIds: string[],
|
||||||
|
): Promise<{ segments: string[]; mapping: Record<string, string> }> {
|
||||||
|
const system = systemPrompt + batchInstructions(usedIds);
|
||||||
|
const content = await chat(system, JSON.stringify({ segments: texts }));
|
||||||
|
const parsed = JSON.parse(content) as { segments?: string[]; mapping?: Record<string, string> };
|
||||||
|
if (!Array.isArray(parsed.segments) || typeof parsed.mapping !== 'object' || !parsed.mapping) {
|
||||||
|
throw new Error('LLM_BATCH_BAD_SHAPE');
|
||||||
|
}
|
||||||
|
return { segments: parsed.segments, mapping: parsed.mapping };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
}
|
||||||
67
src/types.ts
Normal file
67
src/types.ts
Normal file
@@ -0,0 +1,67 @@
|
|||||||
|
/** Result of an anonymization: text with placeholders + the reverse-mapping table. */
|
||||||
|
export interface AnonymizationResult {
|
||||||
|
/** The text with every detected PII value replaced by a placeholder. */
|
||||||
|
anon: string;
|
||||||
|
/** Placeholder → original value, e.g. `{ "[PER_1.NOM:M]": "Alain JACCARD" }`. */
|
||||||
|
mapping: Record<string, string>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Matches a single placeholder. Covers the rich LLM form `[PER_1.NOM:M]`
|
||||||
|
* and the plain regex-fallback form `[EMAIL_1]`.
|
||||||
|
*/
|
||||||
|
export const PLACEHOLDER_RE = /\[[A-Z]+_\d+(?:\.[A-Z_]+:[^\]]+)?\]/g;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Pluggable LLM backend. Implement this (or use {@link openAICompatibleProvider})
|
||||||
|
* to let the {@link Anonymizer} detect free-form PII such as proper names that
|
||||||
|
* regular expressions cannot reliably catch.
|
||||||
|
*/
|
||||||
|
export interface LlmProvider {
|
||||||
|
/** Whether the provider is ready to be called. When `false`, the Anonymizer skips it. */
|
||||||
|
isConfigured(): boolean;
|
||||||
|
/** Anonymize a single piece of text. */
|
||||||
|
anonymize(text: string): Promise<AnonymizationResult>;
|
||||||
|
/**
|
||||||
|
* Anonymize several segments in ONE call with GLOBAL coreference (same entity →
|
||||||
|
* same id across segments) and reuse of the ids already assigned upstream
|
||||||
|
* (`usedIds`). Returns the anonymized segments (same order/length) plus the
|
||||||
|
* mapping of the NEW entities only.
|
||||||
|
*/
|
||||||
|
anonymizeBatch(
|
||||||
|
texts: string[],
|
||||||
|
usedIds: string[],
|
||||||
|
): Promise<{ segments: string[]; mapping: Record<string, string> }>;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** A structured-PII detector: a tag (e.g. `EMAIL`) and the global regex that finds it. */
|
||||||
|
export interface PatternDef {
|
||||||
|
tag: string;
|
||||||
|
re: RegExp;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Minimal logger sink. Defaults to a no-op; pass your own to capture fallback warnings. */
|
||||||
|
export interface Logger {
|
||||||
|
warn(msg: string): void;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Configuration for {@link Anonymizer}. */
|
||||||
|
export interface AnonymizerConfig {
|
||||||
|
/** LLM backend for free-form PII (names, addresses…). Omit for regex-only mode. */
|
||||||
|
llm?: LlmProvider;
|
||||||
|
/** Structured-PII patterns for the regex fallback. Defaults to {@link presets.swiss}. */
|
||||||
|
patterns?: PatternDef[];
|
||||||
|
/** Heuristic that flags likely proper names so the LLM is consulted. Has a sensible default. */
|
||||||
|
nameHint?: RegExp;
|
||||||
|
/** Where fallback/diagnostic warnings go. Defaults to a no-op. */
|
||||||
|
logger?: Logger;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Streaming de-anonymizer: buffers placeholders that get split across token
|
||||||
|
* boundaries so a partial `[PER_` is never emitted (and never leaks).
|
||||||
|
*/
|
||||||
|
export interface StreamDeanonymizer {
|
||||||
|
push(chunk: string): string;
|
||||||
|
flush(): string;
|
||||||
|
}
|
||||||
124
test/anonymizer.test.ts
Normal file
124
test/anonymizer.test.ts
Normal file
@@ -0,0 +1,124 @@
|
|||||||
|
import { describe, it, expect, vi } from 'vitest';
|
||||||
|
import { Anonymizer, presets, type LlmProvider } from '../src/index.js';
|
||||||
|
|
||||||
|
/** An LLM provider that is configured but always fails → forces the regex fallback. */
|
||||||
|
const failingLlm = (overrides: Partial<LlmProvider> = {}): LlmProvider => ({
|
||||||
|
isConfigured: () => true,
|
||||||
|
anonymize: vi.fn().mockRejectedValue(new Error('LLM_NOT_CONFIGURED')),
|
||||||
|
anonymizeBatch: vi.fn().mockRejectedValue(new Error('LLM_NOT_CONFIGURED')),
|
||||||
|
...overrides,
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('Anonymizer (Swiss preset)', () => {
|
||||||
|
const svc = new Anonymizer({ llm: failingLlm(), patterns: presets.swiss });
|
||||||
|
|
||||||
|
it('skips anonymization (no LLM call) when there is no PII', async () => {
|
||||||
|
const llm = failingLlm();
|
||||||
|
const s = new Anonymizer({ llm, patterns: presets.swiss });
|
||||||
|
const r = await s.anonymize('Explique la différence entre INNER JOIN et LEFT JOIN');
|
||||||
|
expect(r.mapping).toEqual({});
|
||||||
|
expect(r.anon).toContain('INNER JOIN');
|
||||||
|
expect(llm.anonymize).not.toHaveBeenCalled();
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to regex for structured PII (email/phone/AVS)', async () => {
|
||||||
|
const r = await svc.anonymize('Contact: jean@exemple.ch, +41 79 123 45 67, AVS 756.1234.5678.90');
|
||||||
|
expect(r.anon).not.toContain('jean@exemple.ch');
|
||||||
|
expect(r.anon).not.toContain('756.1234.5678.90');
|
||||||
|
expect(Object.values(r.mapping)).toContain('jean@exemple.ch');
|
||||||
|
expect(svc.deanonymize(r.anon, r.mapping)).toContain('jean@exemple.ch');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('deanonymizes a placeholder split across stream tokens (critical)', () => {
|
||||||
|
const mapping = { '[PER_1.NOM:M]': 'Alain JACCARD' };
|
||||||
|
const d = svc.makeStreamDeanonymizer(mapping);
|
||||||
|
let out = '';
|
||||||
|
out += d.push('Bonjour [PER_');
|
||||||
|
out += d.push('1.NOM');
|
||||||
|
out += d.push(':M], ravi');
|
||||||
|
out += d.flush();
|
||||||
|
expect(out).toBe('Bonjour Alain JACCARD, ravi');
|
||||||
|
expect(out).not.toContain('[PER_');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('streams plain text through unchanged', () => {
|
||||||
|
const d = svc.makeStreamDeanonymizer({});
|
||||||
|
expect(d.push('Un INNER JOIN ') + d.push('retourne...') + d.flush()).toBe('Un INNER JOIN retourne...');
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('anonymizeChunks', () => {
|
||||||
|
it('PII-free chunks → no LLM call, returned as-is', async () => {
|
||||||
|
const llm = failingLlm();
|
||||||
|
const s = new Anonymizer({ llm, patterns: presets.swiss });
|
||||||
|
const r = await s.anonymizeChunks(['Un INNER JOIN combine deux tables.'], {});
|
||||||
|
expect(llm.anonymizeBatch).not.toHaveBeenCalled();
|
||||||
|
expect(r.anon[0]).toContain('INNER JOIN');
|
||||||
|
expect(r.mapping).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('reuses the question placeholder for the same person (deterministic, no LLM)', async () => {
|
||||||
|
const llm = failingLlm();
|
||||||
|
const s = new Anonymizer({ llm, patterns: presets.swiss });
|
||||||
|
const seed = { '[PER_1.NOM:M]': 'Alain JACCARD' };
|
||||||
|
const r = await s.anonymizeChunks(['Le dossier de Alain JACCARD est complet.'], seed);
|
||||||
|
expect(llm.anonymizeBatch).not.toHaveBeenCalled();
|
||||||
|
expect(r.anon[0]).toContain('[PER_1.NOM:M]');
|
||||||
|
expect(r.anon[0]).not.toContain('Alain JACCARD');
|
||||||
|
expect(r.mapping).toEqual(seed);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('renumbers a NEW person that collides with the question placeholder', async () => {
|
||||||
|
const anonymizeBatch = vi.fn().mockResolvedValue({
|
||||||
|
segments: ['[PER_1.NOM:M] a signé.'],
|
||||||
|
mapping: { '[PER_1.NOM:M]': 'Bob Martin' },
|
||||||
|
});
|
||||||
|
const s = new Anonymizer({ llm: failingLlm({ anonymizeBatch }), patterns: presets.swiss });
|
||||||
|
const seed = { '[PER_1.NOM:M]': 'Alain JACCARD' };
|
||||||
|
const r = await s.anonymizeChunks(['Bob Martin a signé.'], seed);
|
||||||
|
expect(anonymizeBatch).toHaveBeenCalledTimes(1);
|
||||||
|
expect(r.anon[0]).toContain('[PER_2.NOM:M]');
|
||||||
|
expect(r.mapping['[PER_1.NOM:M]']).toBe('Alain JACCARD');
|
||||||
|
expect(r.mapping['[PER_2.NOM:M]']).toBe('Bob Martin');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('falls back to regex (structured ids) when the LLM is unavailable', async () => {
|
||||||
|
const s = new Anonymizer({ llm: failingLlm(), patterns: presets.swiss });
|
||||||
|
const r = await s.anonymizeChunks(['Contact : jean@exemple.ch'], {});
|
||||||
|
expect(r.anon[0]).not.toContain('jean@exemple.ch');
|
||||||
|
expect(Object.values(r.mapping)).toContain('jean@exemple.ch');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('regex-only mode (no LLM provider)', () => {
|
||||||
|
const s = new Anonymizer({ patterns: presets.swiss });
|
||||||
|
|
||||||
|
it('still anonymizes structured PII without any provider', async () => {
|
||||||
|
const r = await s.anonymize('Écris à jean@exemple.ch');
|
||||||
|
expect(r.anon).not.toContain('jean@exemple.ch');
|
||||||
|
expect(Object.values(r.mapping)).toContain('jean@exemple.ch');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('leaves a bare proper name untouched (no LLM to catch it)', async () => {
|
||||||
|
const r = await s.anonymize('Alain Jaccard a réussi');
|
||||||
|
// The name-hint flags it, but with no LLM the regex fallback finds no structured id.
|
||||||
|
expect(r.anon).toContain('Alain Jaccard');
|
||||||
|
expect(r.mapping).toEqual({});
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('configurable presets', () => {
|
||||||
|
it('generic preset anonymizes an IPv4 address', async () => {
|
||||||
|
const s = new Anonymizer({ patterns: presets.generic });
|
||||||
|
const r = await s.anonymize('Serveur 192.168.1.42 indisponible');
|
||||||
|
expect(r.anon).not.toContain('192.168.1.42');
|
||||||
|
expect(Object.values(r.mapping)).toContain('192.168.1.42');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('accepts a fully custom pattern set', async () => {
|
||||||
|
const s = new Anonymizer({ patterns: [{ tag: 'TICKET', re: /\bJIRA-\d+\b/g }] });
|
||||||
|
const r = await s.anonymize('Voir JIRA-123');
|
||||||
|
expect(r.anon).toContain('[TICKET_1]');
|
||||||
|
expect(r.mapping['[TICKET_1]']).toBe('JIRA-123');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
});
|
||||||
58
test/openai-compatible.test.ts
Normal file
58
test/openai-compatible.test.ts
Normal file
@@ -0,0 +1,58 @@
|
|||||||
|
import { describe, it, expect, vi, afterEach } from 'vitest';
|
||||||
|
import { openAICompatibleProvider } from '../src/index.js';
|
||||||
|
|
||||||
|
const opts = { baseUrl: 'https://api.example.com/v1', apiKey: 'k', model: 'm' };
|
||||||
|
|
||||||
|
function mockFetchJson(content: unknown) {
|
||||||
|
return vi.fn().mockResolvedValue({
|
||||||
|
ok: true,
|
||||||
|
json: async () => ({ choices: [{ message: { content: JSON.stringify(content) } }] }),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
afterEach(() => {
|
||||||
|
vi.unstubAllGlobals();
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('openAICompatibleProvider', () => {
|
||||||
|
it('isConfigured() reflects whether baseUrl/apiKey/model are present', () => {
|
||||||
|
expect(openAICompatibleProvider(opts).isConfigured()).toBe(true);
|
||||||
|
expect(openAICompatibleProvider({ ...opts, apiKey: '' }).isConfigured()).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('anonymize() parses {texte_anonymise, mapping} and posts temperature 0 + json_object', async () => {
|
||||||
|
const fetchMock = mockFetchJson({ texte_anonymise: '[EMAIL_1]', mapping: { '[EMAIL_1]': 'a@b.ch' } });
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
|
||||||
|
const r = await openAICompatibleProvider(opts).anonymize('a@b.ch');
|
||||||
|
expect(r).toEqual({ anon: '[EMAIL_1]', mapping: { '[EMAIL_1]': 'a@b.ch' } });
|
||||||
|
|
||||||
|
const [url, init] = fetchMock.mock.calls[0];
|
||||||
|
expect(url).toBe('https://api.example.com/v1/chat/completions');
|
||||||
|
const body = JSON.parse(init.body);
|
||||||
|
expect(body.temperature).toBe(0);
|
||||||
|
expect(body.response_format).toEqual({ type: 'json_object' });
|
||||||
|
expect(init.headers.Authorization).toBe('Bearer k');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('anonymizeBatch() returns {segments, mapping} and includes used ids in the prompt', async () => {
|
||||||
|
const fetchMock = mockFetchJson({ segments: ['[PER_2.NOM:M]'], mapping: { '[PER_2.NOM:M]': 'Bob' } });
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
|
||||||
|
const r = await openAICompatibleProvider(opts).anonymizeBatch(['Bob'], ['[PER_1.NOM:M]']);
|
||||||
|
expect(r.segments).toEqual(['[PER_2.NOM:M]']);
|
||||||
|
|
||||||
|
const system = JSON.parse(fetchMock.mock.calls[0][1].body).messages[0].content;
|
||||||
|
expect(system).toContain('[PER_1.NOM:M]');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('throws on a non-OK HTTP status', async () => {
|
||||||
|
vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: false, status: 500 }));
|
||||||
|
await expect(openAICompatibleProvider(opts).anonymize('x')).rejects.toThrow('LLM_HTTP_500');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('throws on a malformed response shape', async () => {
|
||||||
|
vi.stubGlobal('fetch', mockFetchJson({ wrong: true }));
|
||||||
|
await expect(openAICompatibleProvider(opts).anonymize('x')).rejects.toThrow('LLM_BAD_SHAPE');
|
||||||
|
});
|
||||||
|
});
|
||||||
18
tsconfig.json
Normal file
18
tsconfig.json
Normal file
@@ -0,0 +1,18 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"target": "ES2021",
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "Bundler",
|
||||||
|
"lib": ["ES2021", "DOM"],
|
||||||
|
"declaration": true,
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": true,
|
||||||
|
"noUnusedParameters": true,
|
||||||
|
"noFallthroughCasesInSwitch": true,
|
||||||
|
"esModuleInterop": true,
|
||||||
|
"skipLibCheck": true,
|
||||||
|
"forceConsistentCasingInFileNames": true,
|
||||||
|
"verbatimModuleSyntax": true
|
||||||
|
},
|
||||||
|
"include": ["src/**/*.ts", "test/**/*.ts"]
|
||||||
|
}
|
||||||
11
tsup.config.ts
Normal file
11
tsup.config.ts
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
import { defineConfig } from 'tsup';
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
entry: ['src/index.ts'],
|
||||||
|
format: ['esm', 'cjs'],
|
||||||
|
dts: true,
|
||||||
|
clean: true,
|
||||||
|
sourcemap: true,
|
||||||
|
target: 'es2021',
|
||||||
|
outExtension: ({ format }) => ({ js: format === 'cjs' ? '.cjs' : '.js' }),
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user