feat: harden generic preset, add LLM retry, and finish tooling/docs
- PatternDef.validate + Luhn-gated credit-card detection; de-overlap the generic phone/date/IP patterns; presets.swiss unchanged (production behavior) - strip g/y flags from nameHint so .test() is stateless (latent footgun) - openAICompatibleProvider: bounded retry on transient failures (network / timeout / 429 / 5xx), configurable via retries + retryDelayMs - eslint + prettier + vitest coverage (97%); CI runs lint/format/coverage - docs: README badges + new-option docs, SECURITY.md, issue/PR templates 26 tests passing; build emits ESM+CJS+types. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
35
.github/ISSUE_TEMPLATE/bug_report.md
vendored
Normal file
35
.github/ISSUE_TEMPLATE/bug_report.md
vendored
Normal file
@@ -0,0 +1,35 @@
|
|||||||
|
---
|
||||||
|
name: Bug report
|
||||||
|
about: Report incorrect behavior (false positive/negative, crash, leak)
|
||||||
|
title: ''
|
||||||
|
labels: bug
|
||||||
|
assignees: ''
|
||||||
|
---
|
||||||
|
|
||||||
|
**⚠️ Security/PII-leak issues:** please do NOT file them here — email security@mobiletic.com (see SECURITY.md).
|
||||||
|
|
||||||
|
## Describe the bug
|
||||||
|
|
||||||
|
A clear description of what's wrong.
|
||||||
|
|
||||||
|
## To reproduce
|
||||||
|
|
||||||
|
A minimal code sample or failing test:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
// import { Anonymizer, presets } from '@mobiletic/anonymizer';
|
||||||
|
```
|
||||||
|
|
||||||
|
## Expected behavior
|
||||||
|
|
||||||
|
What you expected to happen.
|
||||||
|
|
||||||
|
## Environment
|
||||||
|
|
||||||
|
- `@mobiletic/anonymizer` version:
|
||||||
|
- Node.js version:
|
||||||
|
- Using an LLM provider? (which endpoint/model, or regex-only):
|
||||||
|
|
||||||
|
## Additional context
|
||||||
|
|
||||||
|
Anything else (input text shape, custom patterns, …). Redact any real PII before posting.
|
||||||
23
.github/ISSUE_TEMPLATE/feature_request.md
vendored
Normal file
23
.github/ISSUE_TEMPLATE/feature_request.md
vendored
Normal file
@@ -0,0 +1,23 @@
|
|||||||
|
---
|
||||||
|
name: Feature request
|
||||||
|
about: Suggest a pattern preset, provider, or capability
|
||||||
|
title: ''
|
||||||
|
labels: enhancement
|
||||||
|
assignees: ''
|
||||||
|
---
|
||||||
|
|
||||||
|
## What problem does this solve?
|
||||||
|
|
||||||
|
The use case or gap you're hitting (e.g. "no preset for German tax IDs", "want an Anthropic provider").
|
||||||
|
|
||||||
|
## Proposed solution
|
||||||
|
|
||||||
|
What you'd like to see. For a new preset, list the identifier(s) and an example value (synthetic, no real PII).
|
||||||
|
|
||||||
|
## Alternatives considered
|
||||||
|
|
||||||
|
Anything you've tried or other approaches.
|
||||||
|
|
||||||
|
## Additional context
|
||||||
|
|
||||||
|
Links, references, or examples.
|
||||||
19
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
19
.github/PULL_REQUEST_TEMPLATE.md
vendored
Normal file
@@ -0,0 +1,19 @@
|
|||||||
|
# Description
|
||||||
|
|
||||||
|
<!-- What does this PR change, and why? -->
|
||||||
|
|
||||||
|
## Type of change
|
||||||
|
|
||||||
|
- [ ] Bug fix
|
||||||
|
- [ ] New feature (pattern preset, provider, …)
|
||||||
|
- [ ] Documentation
|
||||||
|
- [ ] Tooling / chore
|
||||||
|
|
||||||
|
## Checklist
|
||||||
|
|
||||||
|
- [ ] Tests added or updated for the change
|
||||||
|
- [ ] `npm test` passes
|
||||||
|
- [ ] `npm run typecheck` passes
|
||||||
|
- [ ] `npm run lint` and `npm run format:check` pass
|
||||||
|
- [ ] Docs / CHANGELOG updated if behavior or the public API changed
|
||||||
|
- [ ] No runtime dependencies were added to the core
|
||||||
4
.github/workflows/ci.yml
vendored
4
.github/workflows/ci.yml
vendored
@@ -19,6 +19,8 @@ jobs:
|
|||||||
node-version: ${{ matrix.node-version }}
|
node-version: ${{ matrix.node-version }}
|
||||||
cache: npm
|
cache: npm
|
||||||
- run: npm ci
|
- run: npm ci
|
||||||
|
- run: npm run lint
|
||||||
|
- run: npm run format:check
|
||||||
- run: npm run typecheck
|
- run: npm run typecheck
|
||||||
- run: npm test
|
- run: npm run test:coverage
|
||||||
- run: npm run build
|
- run: npm run build
|
||||||
|
|||||||
4
.prettierignore
Normal file
4
.prettierignore
Normal file
@@ -0,0 +1,4 @@
|
|||||||
|
dist
|
||||||
|
coverage
|
||||||
|
node_modules
|
||||||
|
package-lock.json
|
||||||
7
.prettierrc
Normal file
7
.prettierrc
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"semi": true,
|
||||||
|
"singleQuote": true,
|
||||||
|
"trailingComma": "all",
|
||||||
|
"printWidth": 110,
|
||||||
|
"tabWidth": 2
|
||||||
|
}
|
||||||
@@ -15,3 +15,8 @@ All notable changes to this project are documented here. The format is based on
|
|||||||
- `openAICompatibleProvider` — pluggable LLM detection over any OpenAI-compatible Chat Completions API.
|
- `openAICompatibleProvider` — pluggable LLM detection over any OpenAI-compatible Chat Completions API.
|
||||||
- `presets.swiss` and `presets.generic` regex pattern sets; fully configurable custom patterns.
|
- `presets.swiss` and `presets.generic` regex pattern sets; fully configurable custom patterns.
|
||||||
- Regex-only mode (no LLM provider required).
|
- Regex-only mode (no LLM provider required).
|
||||||
|
- `PatternDef.validate` — optional second-stage predicate to cut false positives; `presets.generic` uses
|
||||||
|
it for a Luhn check on credit-card candidates, and de-overlaps its phone/date/IP patterns.
|
||||||
|
- `openAICompatibleProvider` retries transient failures (network/timeout/429/5xx) via `retries` and
|
||||||
|
`retryDelayMs` options; non-transient 4xx and malformed responses are not retried.
|
||||||
|
- Hardened the `nameHint` heuristic: `g`/`y` flags are stripped internally so `.test()` is stateless.
|
||||||
|
|||||||
@@ -9,9 +9,12 @@ new pattern presets, additional LLM providers, docs, and tests.
|
|||||||
git clone https://github.com/mobiletic/anonymizer.git
|
git clone https://github.com/mobiletic/anonymizer.git
|
||||||
cd anonymizer
|
cd anonymizer
|
||||||
npm install
|
npm install
|
||||||
npm test # run the test suite (vitest)
|
npm test # run the test suite (vitest)
|
||||||
npm run typecheck # tsc --noEmit
|
npm run test:coverage # tests + coverage report
|
||||||
npm run build # tsup → dist/ (ESM + CJS + types)
|
npm run typecheck # tsc --noEmit
|
||||||
|
npm run lint # eslint
|
||||||
|
npm run format # prettier --write (or `npm run format:check` to verify)
|
||||||
|
npm run build # tsup → dist/ (ESM + CJS + types)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Guidelines
|
## Guidelines
|
||||||
|
|||||||
30
README.md
30
README.md
@@ -1,5 +1,9 @@
|
|||||||
# @mobiletic/anonymizer
|
# @mobiletic/anonymizer
|
||||||
|
|
||||||
|
[](https://github.com/mobiletic/anonymizer/actions/workflows/ci.yml)
|
||||||
|
[](https://www.npmjs.com/package/@mobiletic/anonymizer)
|
||||||
|
[](./LICENSE)
|
||||||
|
|
||||||
Framework-agnostic **PII anonymization & pseudonymization** for TypeScript/JavaScript.
|
Framework-agnostic **PII anonymization & pseudonymization** for TypeScript/JavaScript.
|
||||||
|
|
||||||
It replaces personal data in text with stable placeholders before the text leaves your trust boundary
|
It replaces personal data in text with stable placeholders before the text leaves your trust boundary
|
||||||
@@ -64,6 +68,11 @@ const { anon, mapping } = await anonymizer.anonymize('Le dossier de Alain Jaccar
|
|||||||
If the LLM call fails, times out, or returns an invalid shape, the anonymizer automatically falls back to
|
If the LLM call fails, times out, or returns an invalid shape, the anonymizer automatically falls back to
|
||||||
the regex engine — it never throws on a provider failure.
|
the regex engine — it never throws on a provider failure.
|
||||||
|
|
||||||
|
`openAICompatibleProvider` retries **transient** failures (network error, timeout, HTTP 429/5xx) before
|
||||||
|
giving up; 4xx and malformed responses are not retried. Tune with `timeoutMs` (per attempt, default 3000),
|
||||||
|
`retries` (default 1 → 2 attempts), and `retryDelayMs` (linear backoff, default 250). Worst-case latency
|
||||||
|
is `(retries + 1) × timeoutMs`, so keep `retries` low on latency-sensitive paths.
|
||||||
|
|
||||||
### Streaming de-anonymization
|
### Streaming de-anonymization
|
||||||
|
|
||||||
When you stream an LLM answer back to a user, restore real values without ever emitting a half-written
|
When you stream an LLM answer back to a user, restore real values without ever emitting a half-written
|
||||||
@@ -102,7 +111,7 @@ new Anonymizer({
|
|||||||
```ts
|
```ts
|
||||||
import { presets } from '@mobiletic/anonymizer';
|
import { presets } from '@mobiletic/anonymizer';
|
||||||
|
|
||||||
presets.swiss; // AVS, IBAN CH, EMAIL, Swiss phone, DATE
|
presets.swiss; // AVS, IBAN CH, EMAIL, Swiss phone, DATE
|
||||||
presets.generic; // EMAIL, IBAN, credit card, IPv4, phone, DATE
|
presets.generic; // EMAIL, IBAN, credit card, IPv4, phone, DATE
|
||||||
|
|
||||||
// Compose / extend:
|
// Compose / extend:
|
||||||
@@ -112,6 +121,17 @@ const patterns = [
|
|||||||
];
|
];
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Each pattern may carry an optional `validate(match) => boolean` second stage — a match is only redacted
|
||||||
|
if it passes. `presets.generic` uses it for a [Luhn](https://en.wikipedia.org/wiki/Luhn_algorithm) check
|
||||||
|
so arbitrary long digit runs aren't mistaken for credit cards:
|
||||||
|
|
||||||
|
```ts
|
||||||
|
{ tag: 'CREDIT_CARD', re: /\b\d(?:[ -]?\d){12,18}\b/g, validate: luhnValid }
|
||||||
|
```
|
||||||
|
|
||||||
|
> The `generic` preset is a **best-effort** starting point — broad patterns (phone, date, card) can
|
||||||
|
> overlap. For production use, prefer a locale-specific preset (`presets.swiss`) or your own patterns.
|
||||||
|
|
||||||
### Custom LLM provider
|
### Custom LLM provider
|
||||||
|
|
||||||
Implement `LlmProvider` to use any backend (Anthropic, a local model, a rules engine…):
|
Implement `LlmProvider` to use any backend (Anthropic, a local model, a rules engine…):
|
||||||
@@ -121,8 +141,12 @@ import type { LlmProvider } from '@mobiletic/anonymizer';
|
|||||||
|
|
||||||
const myProvider: LlmProvider = {
|
const myProvider: LlmProvider = {
|
||||||
isConfigured: () => true,
|
isConfigured: () => true,
|
||||||
async anonymize(text) { /* return { anon, mapping } */ },
|
async anonymize(text) {
|
||||||
async anonymizeBatch(texts, usedIds) { /* return { segments, mapping } */ },
|
/* return { anon, mapping } */
|
||||||
|
},
|
||||||
|
async anonymizeBatch(texts, usedIds) {
|
||||||
|
/* return { segments, mapping } */
|
||||||
|
},
|
||||||
};
|
};
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|||||||
28
SECURITY.md
Normal file
28
SECURITY.md
Normal file
@@ -0,0 +1,28 @@
|
|||||||
|
# Security Policy
|
||||||
|
|
||||||
|
## Reporting a vulnerability
|
||||||
|
|
||||||
|
This library handles personal data, so we take security and privacy issues seriously — especially any
|
||||||
|
path that could cause PII to **leak** (e.g. a placeholder that isn't restored, or sensitive text reaching
|
||||||
|
a downstream service un-anonymized).
|
||||||
|
|
||||||
|
**Please do not open a public issue for security problems.** Instead, email **security@mobiletic.com**
|
||||||
|
with:
|
||||||
|
|
||||||
|
- a description of the issue and its impact,
|
||||||
|
- steps to reproduce (a minimal code sample or failing test is ideal),
|
||||||
|
- the package version and Node.js version.
|
||||||
|
|
||||||
|
We aim to acknowledge reports within a few business days and will keep you updated on remediation. Once a
|
||||||
|
fix is released, we're happy to credit you (unless you prefer to remain anonymous).
|
||||||
|
|
||||||
|
## Supported versions
|
||||||
|
|
||||||
|
This project is pre-1.0; security fixes land on the latest published release. We recommend always running
|
||||||
|
the most recent version.
|
||||||
|
|
||||||
|
## Scope & disclaimer
|
||||||
|
|
||||||
|
This library is a **best-effort** pseudonymization aid, not a guarantee of regulatory compliance. LLM and
|
||||||
|
regex detection can miss or misclassify data. Validate against your own requirements (nLPD, GDPR, HIPAA, …)
|
||||||
|
before relying on it for regulated data.
|
||||||
15
eslint.config.js
Normal file
15
eslint.config.js
Normal file
@@ -0,0 +1,15 @@
|
|||||||
|
import js from '@eslint/js';
|
||||||
|
import tseslint from 'typescript-eslint';
|
||||||
|
import prettier from 'eslint-config-prettier';
|
||||||
|
|
||||||
|
export default tseslint.config(
|
||||||
|
{ ignores: ['dist/**', 'coverage/**', 'node_modules/**'] },
|
||||||
|
js.configs.recommended,
|
||||||
|
...tseslint.configs.recommended,
|
||||||
|
prettier,
|
||||||
|
{
|
||||||
|
rules: {
|
||||||
|
'@typescript-eslint/no-non-null-assertion': 'off',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
);
|
||||||
1622
package-lock.json
generated
1622
package-lock.json
generated
File diff suppressed because it is too large
Load Diff
10
package.json
10
package.json
@@ -46,12 +46,22 @@
|
|||||||
"build": "tsup",
|
"build": "tsup",
|
||||||
"test": "vitest run",
|
"test": "vitest run",
|
||||||
"test:watch": "vitest",
|
"test:watch": "vitest",
|
||||||
|
"test:coverage": "vitest run --coverage",
|
||||||
"typecheck": "tsc --noEmit",
|
"typecheck": "tsc --noEmit",
|
||||||
|
"lint": "eslint .",
|
||||||
|
"format": "prettier --write .",
|
||||||
|
"format:check": "prettier --check .",
|
||||||
"prepublishOnly": "npm run build"
|
"prepublishOnly": "npm run build"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
|
"@eslint/js": "^9.0.0",
|
||||||
|
"@vitest/coverage-v8": "^1.6.0",
|
||||||
|
"eslint": "^9.0.0",
|
||||||
|
"eslint-config-prettier": "^9.1.0",
|
||||||
|
"prettier": "^3.2.0",
|
||||||
"tsup": "^8.0.0",
|
"tsup": "^8.0.0",
|
||||||
"typescript": "^5.4.0",
|
"typescript": "^5.4.0",
|
||||||
|
"typescript-eslint": "^8.0.0",
|
||||||
"vitest": "^1.6.0"
|
"vitest": "^1.6.0"
|
||||||
},
|
},
|
||||||
"publishConfig": {
|
"publishConfig": {
|
||||||
|
|||||||
@@ -14,6 +14,15 @@ const NOOP_LOGGER: Logger = { warn: () => {} };
|
|||||||
// Likely proper-name heuristic (two capitalized words), in addition to the structured fallback.
|
// Likely proper-name heuristic (two capitalized words), in addition to the structured fallback.
|
||||||
const DEFAULT_NAME_HINT = /\b[A-ZÀ-Ý][a-zà-ÿ]+\s+[A-ZÀ-Ý][a-zà-ÿ]+\b/;
|
const DEFAULT_NAME_HINT = /\b[A-ZÀ-Ý][a-zà-ÿ]+\s+[A-ZÀ-Ý][a-zà-ÿ]+\b/;
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Return a stateless copy of a regex: a global/sticky (`g`/`y`) regex carries a
|
||||||
|
* mutable `lastIndex`, which makes repeated `.test()` calls return alternating
|
||||||
|
* results. The name-hint is used with `.test()`, so we strip those flags.
|
||||||
|
*/
|
||||||
|
function stateless(re: RegExp): RegExp {
|
||||||
|
return re.global || re.sticky ? new RegExp(re.source, re.flags.replace(/[gy]/g, '')) : re;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Framework-agnostic anonymization engine.
|
* Framework-agnostic anonymization engine.
|
||||||
*
|
*
|
||||||
@@ -34,7 +43,7 @@ export class Anonymizer {
|
|||||||
constructor(config: AnonymizerConfig = {}) {
|
constructor(config: AnonymizerConfig = {}) {
|
||||||
this.llm = config.llm;
|
this.llm = config.llm;
|
||||||
this.fallback = new RegexFallback(config.patterns ?? presets.swiss);
|
this.fallback = new RegexFallback(config.patterns ?? presets.swiss);
|
||||||
this.nameHint = config.nameHint ?? DEFAULT_NAME_HINT;
|
this.nameHint = stateless(config.nameHint ?? DEFAULT_NAME_HINT);
|
||||||
this.logger = config.logger ?? NOOP_LOGGER;
|
this.logger = config.logger ?? NOOP_LOGGER;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -16,8 +16,10 @@ export class RegexFallback {
|
|||||||
const counters: Record<string, number> = {};
|
const counters: Record<string, number> = {};
|
||||||
let anon = text;
|
let anon = text;
|
||||||
|
|
||||||
for (const { tag, re } of this.patterns) {
|
for (const { tag, re, validate } of this.patterns) {
|
||||||
anon = anon.replace(re, (match) => {
|
anon = anon.replace(re, (match) => {
|
||||||
|
// A validator can reject a regex match (e.g. Luhn) → leave the text as-is.
|
||||||
|
if (validate && !validate(match)) return match;
|
||||||
// Reuse the same placeholder for an identical value (consistency).
|
// Reuse the same placeholder for an identical value (consistency).
|
||||||
const existing = Object.entries(mapping).find(([, v]) => v === match);
|
const existing = Object.entries(mapping).find(([, v]) => v === match);
|
||||||
if (existing) return existing[0];
|
if (existing) return existing[0];
|
||||||
@@ -32,9 +34,16 @@ export class RegexFallback {
|
|||||||
|
|
||||||
/** Fast pre-check: is there any structured identifier worth anonymizing? */
|
/** Fast pre-check: is there any structured identifier worth anonymizing? */
|
||||||
hasPii(text: string): boolean {
|
hasPii(text: string): boolean {
|
||||||
return this.patterns.some(({ re }) => {
|
return this.patterns.some(({ re, validate }) => {
|
||||||
re.lastIndex = 0;
|
re.lastIndex = 0;
|
||||||
return re.test(text);
|
if (!validate) return re.test(text);
|
||||||
|
// With a validator, only a match that passes it counts as PII.
|
||||||
|
let m: RegExpExecArray | null;
|
||||||
|
while ((m = re.exec(text)) !== null) {
|
||||||
|
if (validate(m[0])) return true;
|
||||||
|
if (m.index === re.lastIndex) re.lastIndex++; // guard against zero-width matches
|
||||||
|
}
|
||||||
|
return false;
|
||||||
});
|
});
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -12,17 +12,43 @@ const swiss: PatternDef[] = [
|
|||||||
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
||||||
];
|
];
|
||||||
|
|
||||||
|
/** Luhn check on a credit-card candidate (ignores spaces/dashes); 13–19 digits. */
|
||||||
|
function luhnValid(value: string): boolean {
|
||||||
|
const digits = value.replace(/\D/g, '');
|
||||||
|
if (digits.length < 13 || digits.length > 19) return false;
|
||||||
|
let sum = 0;
|
||||||
|
let double = false;
|
||||||
|
for (let i = digits.length - 1; i >= 0; i--) {
|
||||||
|
let d = digits.charCodeAt(i) - 48; // '0' === 48
|
||||||
|
if (double) {
|
||||||
|
d *= 2;
|
||||||
|
if (d > 9) d -= 9;
|
||||||
|
}
|
||||||
|
sum += d;
|
||||||
|
double = !double;
|
||||||
|
}
|
||||||
|
return sum % 10 === 0;
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Generic preset — locale-agnostic identifiers useful as a starting point
|
* Generic preset — locale-agnostic identifiers useful as a starting point
|
||||||
* anywhere. Extend or compose with your own {@link PatternDef}s as needed.
|
* anywhere. Best-effort: extend or compose with your own {@link PatternDef}s.
|
||||||
|
*
|
||||||
|
* Order matters (patterns run in sequence on the already-anonymized text):
|
||||||
|
* specific identifiers claim their matches before broader ones, and dates are
|
||||||
|
* claimed before phone numbers so they don't get mis-tagged as `TEL`.
|
||||||
*/
|
*/
|
||||||
const generic: PatternDef[] = [
|
const generic: PatternDef[] = [
|
||||||
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g },
|
{ tag: 'EMAIL', re: /\b[\w.+-]+@[\w-]+\.[\w.-]+\b/g },
|
||||||
{ tag: 'IBAN', re: /\b[A-Z]{2}\d{2}[0-9A-Z]{11,30}\b/g },
|
{ tag: 'IBAN', re: /\b[A-Z]{2}\d{2}[0-9A-Z]{11,30}\b/g },
|
||||||
{ tag: 'CREDIT_CARD', re: /\b(?:\d[ -]?){13,16}\b/g },
|
|
||||||
{ tag: 'IPV4', re: /\b(?:\d{1,3}\.){3}\d{1,3}\b/g },
|
{ tag: 'IPV4', re: /\b(?:\d{1,3}\.){3}\d{1,3}\b/g },
|
||||||
{ tag: 'TEL', re: /\b(?:\+?\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]?){2,4}\d{2,4}\b/g },
|
// 13–19 digits in optional space/dash groups, gated by Luhn to avoid eating
|
||||||
|
// arbitrary long digit runs (account numbers, ids).
|
||||||
|
{ tag: 'CREDIT_CARD', re: /\b\d(?:[ -]?\d){12,18}\b/g, validate: luhnValid },
|
||||||
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
{ tag: 'DATE', re: /\b\d{1,2}[./-]\d{1,2}[./-]\d{2,4}\b/g },
|
||||||
|
// Requires grouped/separated digits (or a leading +country) so plain integers
|
||||||
|
// aren't mistaken for phone numbers.
|
||||||
|
{ tag: 'TEL', re: /(?:\+\d{1,3}[\s.-]?)?(?:\(?\d{2,4}\)?[\s.-]){2,3}\d{2,4}/g },
|
||||||
];
|
];
|
||||||
|
|
||||||
/** Built-in pattern sets for the regex fallback. */
|
/** Built-in pattern sets for the regex fallback. */
|
||||||
|
|||||||
@@ -28,14 +28,26 @@ export interface OpenAICompatibleOptions {
|
|||||||
apiKey: string;
|
apiKey: string;
|
||||||
/** Model id, e.g. `gpt-4o-mini` or `gemma-3-...`. */
|
/** Model id, e.g. `gpt-4o-mini` or `gemma-3-...`. */
|
||||||
model: string;
|
model: string;
|
||||||
/** Per-request timeout in milliseconds (default 3000). */
|
/** Per-request timeout in milliseconds, per attempt (default 3000). */
|
||||||
timeoutMs?: number;
|
timeoutMs?: number;
|
||||||
|
/**
|
||||||
|
* Extra retries on TRANSIENT failures only (network error, timeout, HTTP 429/5xx).
|
||||||
|
* Default 1 (→ up to 2 attempts). Worst-case latency is `(retries + 1) × timeoutMs`.
|
||||||
|
*/
|
||||||
|
retries?: number;
|
||||||
|
/** Linear backoff between attempts in milliseconds (delay = attempt × this). Default 250. */
|
||||||
|
retryDelayMs?: number;
|
||||||
/** Override the system prompt (e.g. for another language or regulation). */
|
/** Override the system prompt (e.g. for another language or regulation). */
|
||||||
systemPrompt?: string;
|
systemPrompt?: string;
|
||||||
/** Extra instructions appended to the system prompt in batch mode. */
|
/** Extra instructions appended to the system prompt in batch mode. */
|
||||||
batchInstructions?: (usedIds: string[]) => string;
|
batchInstructions?: (usedIds: string[]) => string;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
const sleep = (ms: number): Promise<void> => new Promise((resolve) => setTimeout(resolve, ms));
|
||||||
|
|
||||||
|
/** A transport error that the caller may retry (network / timeout / 429 / 5xx). */
|
||||||
|
class TransientError extends Error {}
|
||||||
|
|
||||||
const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
||||||
'\n\nMODE LOT (segments) :' +
|
'\n\nMODE LOT (segments) :' +
|
||||||
'\n- ENTRÉE : un objet JSON {"segments": ["…", "…"]}.' +
|
'\n- ENTRÉE : un objet JSON {"segments": ["…", "…"]}.' +
|
||||||
@@ -54,17 +66,20 @@ const DEFAULT_BATCH_INSTRUCTIONS = (usedIds: string[]): string =>
|
|||||||
*/
|
*/
|
||||||
export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProvider {
|
export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProvider {
|
||||||
const timeout = opts.timeoutMs ?? 3000;
|
const timeout = opts.timeoutMs ?? 3000;
|
||||||
|
const retries = Math.max(0, opts.retries ?? 1);
|
||||||
|
const retryDelayMs = opts.retryDelayMs ?? 250;
|
||||||
const systemPrompt = opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
|
const systemPrompt = opts.systemPrompt ?? DEFAULT_SYSTEM_PROMPT;
|
||||||
const batchInstructions = opts.batchInstructions ?? DEFAULT_BATCH_INSTRUCTIONS;
|
const batchInstructions = opts.batchInstructions ?? DEFAULT_BATCH_INSTRUCTIONS;
|
||||||
|
|
||||||
const isConfigured = (): boolean => !!(opts.baseUrl && opts.apiKey && opts.model);
|
const isConfigured = (): boolean => !!(opts.baseUrl && opts.apiKey && opts.model);
|
||||||
|
|
||||||
async function chat(system: string, user: string): Promise<string> {
|
/** One HTTP attempt. Throws {@link TransientError} for retryable failures. */
|
||||||
if (!isConfigured()) throw new Error('LLM_NOT_CONFIGURED');
|
async function attempt(system: string, user: string): Promise<string> {
|
||||||
const controller = new AbortController();
|
const controller = new AbortController();
|
||||||
const timer = setTimeout(() => controller.abort(), timeout);
|
const timer = setTimeout(() => controller.abort(), timeout);
|
||||||
|
let res: Response;
|
||||||
try {
|
try {
|
||||||
const res = await fetch(`${opts.baseUrl}/chat/completions`, {
|
res = await fetch(`${opts.baseUrl}/chat/completions`, {
|
||||||
method: 'POST',
|
method: 'POST',
|
||||||
headers: {
|
headers: {
|
||||||
'Content-Type': 'application/json',
|
'Content-Type': 'application/json',
|
||||||
@@ -81,12 +96,34 @@ export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProv
|
|||||||
}),
|
}),
|
||||||
signal: controller.signal,
|
signal: controller.signal,
|
||||||
});
|
});
|
||||||
if (!res.ok) throw new Error(`LLM_HTTP_${res.status}`);
|
} catch (err) {
|
||||||
const data = (await res.json()) as { choices?: Array<{ message?: { content?: string } }> };
|
// Network failure or timeout/abort → retryable.
|
||||||
return data?.choices?.[0]?.message?.content ?? '';
|
throw new TransientError(`LLM_FETCH_FAILED: ${(err as Error).message}`);
|
||||||
} finally {
|
} finally {
|
||||||
clearTimeout(timer);
|
clearTimeout(timer);
|
||||||
}
|
}
|
||||||
|
if (!res.ok) {
|
||||||
|
// 429 + 5xx are transient; other 4xx are not (won't fix on retry).
|
||||||
|
if (res.status === 429 || res.status >= 500) throw new TransientError(`LLM_HTTP_${res.status}`);
|
||||||
|
throw new Error(`LLM_HTTP_${res.status}`);
|
||||||
|
}
|
||||||
|
const data = (await res.json()) as { choices?: Array<{ message?: { content?: string } }> };
|
||||||
|
return data?.choices?.[0]?.message?.content ?? '';
|
||||||
|
}
|
||||||
|
|
||||||
|
async function chat(system: string, user: string): Promise<string> {
|
||||||
|
if (!isConfigured()) throw new Error('LLM_NOT_CONFIGURED');
|
||||||
|
let lastErr: unknown;
|
||||||
|
for (let i = 0; i <= retries; i++) {
|
||||||
|
try {
|
||||||
|
return await attempt(system, user);
|
||||||
|
} catch (err) {
|
||||||
|
lastErr = err;
|
||||||
|
if (!(err instanceof TransientError) || i === retries) throw err;
|
||||||
|
if (retryDelayMs > 0) await sleep(retryDelayMs * (i + 1));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
throw lastErr; // unreachable, but keeps the type checker happy
|
||||||
}
|
}
|
||||||
|
|
||||||
return {
|
return {
|
||||||
@@ -95,7 +132,11 @@ export function openAICompatibleProvider(opts: OpenAICompatibleOptions): LlmProv
|
|||||||
async anonymize(text: string): Promise<AnonymizationResult> {
|
async anonymize(text: string): Promise<AnonymizationResult> {
|
||||||
const content = await chat(systemPrompt, text);
|
const content = await chat(systemPrompt, text);
|
||||||
const parsed = JSON.parse(content) as { texte_anonymise?: string; mapping?: Record<string, string> };
|
const parsed = JSON.parse(content) as { texte_anonymise?: string; mapping?: Record<string, string> };
|
||||||
if (typeof parsed.texte_anonymise !== 'string' || typeof parsed.mapping !== 'object' || !parsed.mapping) {
|
if (
|
||||||
|
typeof parsed.texte_anonymise !== 'string' ||
|
||||||
|
typeof parsed.mapping !== 'object' ||
|
||||||
|
!parsed.mapping
|
||||||
|
) {
|
||||||
throw new Error('LLM_BAD_SHAPE');
|
throw new Error('LLM_BAD_SHAPE');
|
||||||
}
|
}
|
||||||
return { anon: parsed.texte_anonymise, mapping: parsed.mapping };
|
return { anon: parsed.texte_anonymise, mapping: parsed.mapping };
|
||||||
|
|||||||
13
src/types.ts
13
src/types.ts
@@ -37,7 +37,14 @@ export interface LlmProvider {
|
|||||||
/** A structured-PII detector: a tag (e.g. `EMAIL`) and the global regex that finds it. */
|
/** A structured-PII detector: a tag (e.g. `EMAIL`) and the global regex that finds it. */
|
||||||
export interface PatternDef {
|
export interface PatternDef {
|
||||||
tag: string;
|
tag: string;
|
||||||
|
/** Global (`/…/g`) regex that finds candidate matches. */
|
||||||
re: RegExp;
|
re: RegExp;
|
||||||
|
/**
|
||||||
|
* Optional second-stage check. When present, a regex match is only treated as
|
||||||
|
* PII if `validate(match)` returns true — otherwise the text is left untouched.
|
||||||
|
* Use it to cut false positives (e.g. a Luhn check on credit-card candidates).
|
||||||
|
*/
|
||||||
|
validate?: (match: string) => boolean;
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Minimal logger sink. Defaults to a no-op; pass your own to capture fallback warnings. */
|
/** Minimal logger sink. Defaults to a no-op; pass your own to capture fallback warnings. */
|
||||||
@@ -51,7 +58,11 @@ export interface AnonymizerConfig {
|
|||||||
llm?: LlmProvider;
|
llm?: LlmProvider;
|
||||||
/** Structured-PII patterns for the regex fallback. Defaults to {@link presets.swiss}. */
|
/** Structured-PII patterns for the regex fallback. Defaults to {@link presets.swiss}. */
|
||||||
patterns?: PatternDef[];
|
patterns?: PatternDef[];
|
||||||
/** Heuristic that flags likely proper names so the LLM is consulted. Has a sensible default. */
|
/**
|
||||||
|
* Heuristic that flags likely proper names so the LLM is consulted. Has a
|
||||||
|
* sensible default. Any `g`/`y` flags are stripped internally so the regex is
|
||||||
|
* used statelessly — a global flag would otherwise alternate results.
|
||||||
|
*/
|
||||||
nameHint?: RegExp;
|
nameHint?: RegExp;
|
||||||
/** Where fallback/diagnostic warnings go. Defaults to a no-op. */
|
/** Where fallback/diagnostic warnings go. Defaults to a no-op. */
|
||||||
logger?: Logger;
|
logger?: Logger;
|
||||||
|
|||||||
@@ -46,13 +46,71 @@ describe('openAICompatibleProvider', () => {
|
|||||||
expect(system).toContain('[PER_1.NOM:M]');
|
expect(system).toContain('[PER_1.NOM:M]');
|
||||||
});
|
});
|
||||||
|
|
||||||
it('throws on a non-OK HTTP status', async () => {
|
it('throws on a non-OK HTTP status (no retry when retries: 0)', async () => {
|
||||||
vi.stubGlobal('fetch', vi.fn().mockResolvedValue({ ok: false, status: 500 }));
|
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 500 });
|
||||||
await expect(openAICompatibleProvider(opts).anonymize('x')).rejects.toThrow('LLM_HTTP_500');
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
await expect(openAICompatibleProvider({ ...opts, retries: 0 }).anonymize('x')).rejects.toThrow(
|
||||||
|
'LLM_HTTP_500',
|
||||||
|
);
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||||
});
|
});
|
||||||
|
|
||||||
it('throws on a malformed response shape', async () => {
|
it('throws on a malformed response shape', async () => {
|
||||||
vi.stubGlobal('fetch', mockFetchJson({ wrong: true }));
|
vi.stubGlobal('fetch', mockFetchJson({ wrong: true }));
|
||||||
await expect(openAICompatibleProvider(opts).anonymize('x')).rejects.toThrow('LLM_BAD_SHAPE');
|
await expect(openAICompatibleProvider({ ...opts, retries: 0 }).anonymize('x')).rejects.toThrow(
|
||||||
|
'LLM_BAD_SHAPE',
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('retry/backoff', () => {
|
||||||
|
const retryOpts = { ...opts, retries: 1, retryDelayMs: 0 };
|
||||||
|
|
||||||
|
it('retries once on a transient 500 then succeeds (2 calls)', async () => {
|
||||||
|
const fetchMock = vi
|
||||||
|
.fn()
|
||||||
|
.mockResolvedValueOnce({ ok: false, status: 500 })
|
||||||
|
.mockResolvedValueOnce({
|
||||||
|
ok: true,
|
||||||
|
json: async () => ({
|
||||||
|
choices: [{ message: { content: JSON.stringify({ texte_anonymise: 'ok', mapping: {} }) } }],
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
|
||||||
|
const r = await openAICompatibleProvider(retryOpts).anonymize('x');
|
||||||
|
expect(r.anon).toBe('ok');
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('retries on a network error then succeeds', async () => {
|
||||||
|
const fetchMock = vi
|
||||||
|
.fn()
|
||||||
|
.mockRejectedValueOnce(new Error('ECONNRESET'))
|
||||||
|
.mockResolvedValueOnce({
|
||||||
|
ok: true,
|
||||||
|
json: async () => ({
|
||||||
|
choices: [{ message: { content: JSON.stringify({ texte_anonymise: 'ok', mapping: {} }) } }],
|
||||||
|
}),
|
||||||
|
});
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
|
||||||
|
const r = await openAICompatibleProvider(retryOpts).anonymize('x');
|
||||||
|
expect(r.anon).toBe('ok');
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('does NOT retry a 400 (non-transient)', async () => {
|
||||||
|
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 400 });
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
await expect(openAICompatibleProvider(retryOpts).anonymize('x')).rejects.toThrow('LLM_HTTP_400');
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(1);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('gives up after exhausting retries on persistent 503', async () => {
|
||||||
|
const fetchMock = vi.fn().mockResolvedValue({ ok: false, status: 503 });
|
||||||
|
vi.stubGlobal('fetch', fetchMock);
|
||||||
|
await expect(openAICompatibleProvider(retryOpts).anonymize('x')).rejects.toThrow('LLM_HTTP_503');
|
||||||
|
expect(fetchMock).toHaveBeenCalledTimes(2); // initial + 1 retry
|
||||||
|
});
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
50
test/presets.test.ts
Normal file
50
test/presets.test.ts
Normal file
@@ -0,0 +1,50 @@
|
|||||||
|
import { describe, it, expect } from 'vitest';
|
||||||
|
import { Anonymizer, presets } from '../src/index.js';
|
||||||
|
|
||||||
|
describe('generic preset', () => {
|
||||||
|
const a = new Anonymizer({ patterns: presets.generic });
|
||||||
|
|
||||||
|
it('redacts a Luhn-valid credit card', async () => {
|
||||||
|
const r = await a.anonymize('Carte 4111 1111 1111 1111 acceptée'); // valid test Visa
|
||||||
|
expect(r.anon).not.toContain('4111');
|
||||||
|
expect(Object.values(r.mapping)).toContain('4111 1111 1111 1111');
|
||||||
|
expect(Object.keys(r.mapping)[0]).toMatch(/^\[CREDIT_CARD_1\]$/);
|
||||||
|
});
|
||||||
|
|
||||||
|
it('leaves a Luhn-INVALID digit run intact (not tagged as a card)', async () => {
|
||||||
|
// Unseparated run: no other generic pattern matches it either, so it stays verbatim.
|
||||||
|
const r = await a.anonymize('Ref 1234567890123456 interne'); // fails Luhn
|
||||||
|
expect(r.anon).toContain('1234567890123456');
|
||||||
|
expect(r.mapping).toEqual({});
|
||||||
|
});
|
||||||
|
|
||||||
|
it('redacts an IPv4 address without mangling it as a phone or card', async () => {
|
||||||
|
const r = await a.anonymize('Serveur 192.168.1.42 indisponible');
|
||||||
|
expect(r.mapping).toEqual({ '[IPV4_1]': '192.168.1.42' });
|
||||||
|
expect(a.deanonymize(r.anon, r.mapping)).toContain('192.168.1.42');
|
||||||
|
});
|
||||||
|
|
||||||
|
it('tags a date as DATE (not TEL) and keeps a phone separate', async () => {
|
||||||
|
const r = await a.anonymize('Le 12.03.2024, appelez le +41 22 345 67 89.');
|
||||||
|
const tags = Object.keys(r.mapping).map((p) => p.replace(/_\d+\]$/, ']'));
|
||||||
|
expect(tags).toContain('[DATE]');
|
||||||
|
expect(tags).toContain('[TEL]');
|
||||||
|
expect(r.mapping['[DATE_1]']).toBe('12.03.2024');
|
||||||
|
// round-trip is lossless
|
||||||
|
expect(a.deanonymize(r.anon, r.mapping)).toBe('Le 12.03.2024, appelez le +41 22 345 67 89.');
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
describe('nameHint stability', () => {
|
||||||
|
it('a global-flagged custom nameHint yields the same result across repeated calls', async () => {
|
||||||
|
// A naive `.test()` on a /g regex alternates true/false; the Anonymizer must strip the flag.
|
||||||
|
const a = new Anonymizer({ nameHint: /\bDr\.\s\w+/g, patterns: presets.swiss });
|
||||||
|
const text = 'Rendez-vous avec Dr. Meyer'; // no structured PII, only the name hint
|
||||||
|
const r1 = await a.anonymize(text);
|
||||||
|
const r2 = await a.anonymize(text);
|
||||||
|
const r3 = await a.anonymize(text);
|
||||||
|
// No LLM → nothing is actually redacted, but behavior must be identical every time.
|
||||||
|
expect(r1).toEqual(r2);
|
||||||
|
expect(r2).toEqual(r3);
|
||||||
|
});
|
||||||
|
});
|
||||||
11
vitest.config.ts
Normal file
11
vitest.config.ts
Normal file
@@ -0,0 +1,11 @@
|
|||||||
|
import { defineConfig } from 'vitest/config';
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
test: {
|
||||||
|
coverage: {
|
||||||
|
provider: 'v8',
|
||||||
|
include: ['src/**/*.ts'],
|
||||||
|
reporter: ['text', 'html', 'lcov'],
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user