Skip to content

Redaction Ledgers

A Redaction Ledger is a core security feature of Philter, providing a cryptographically verifiable, tamper-evident, append-only log of the redactions performed in contexts that have the ledger enabled. In an era where data integrity and transparency are paramount, ledgers offer a definitive way to audit and trust the automated redaction process.

By maintaining a verifiable record of what was changed, why it was changed, and how it was changed, Philter empowers your organization to demonstrate compliance with rigorous data privacy standards like HIPAA, GDPR, and CCPA. The chain is tamper-evident and each entry is signed, so an altered entry is detectable and its origin is provable. See What the Ledger Proves. As described under How and When Ledger Entries Are Deleted, entries can still be removed deliberately for data-minimization or lifecycle reasons.

How Redaction Ledgers Work

The Philterd ledger system is built on the principles of cryptographic chaining, similar to a blockchain.

  1. Granular Recording: When a plain-text document is processed in a ledger-enabled context, each identified PII or PHI span is recorded as an individual entry in that document's ledger. PDFs are recorded as a chain without per-redaction entries (see Important Considerations and Limitations).
  2. Cryptographic Chaining: Each ledger entry contains a cryptographic hash of its own data plus the hash of the preceding entry. This creates a "chain of trust."
  3. Tamper Evidence: Because each entry is linked to the previous one, retroactively modifying an entry or removing one from within a chain breaks the cryptographic chain, which validation detects. Whole chains can still be deleted deliberately when deletion is enabled (see How and When Ledger Entries Are Deleted).
  4. Verifiability: This architecture allows you to mathematically prove the integrity of your redaction history at any point in time.

What the Ledger Proves

Two separate guarantees, which fail for different reasons and are reported separately by GET /api/ledger/{documentId}/valid.

The hash chain proves internal consistency. Each entry contains the hash of the previous entry, so altering an entry breaks every link after it. On its own this is not proof of origin: anyone able to write to the database could rewrite an entry, recompute its hash, and relink the entries that follow, producing a chain that verifies perfectly.

Every recorded value is covered by that hash: the owning user and document, the redacted value and its replacement, its position, the document fingerprint, the timestamp, the link to the previous entry, the governing policy name, version and fingerprint, the effective configuration hash, the filename, and the PII type. None can be changed without the entry failing validation. Only the signature and the id of the key that made it sit outside the hash, because the signature is taken over the hash.

How the hash is computed. Each field is written as its length in UTF-8 bytes, a colon, then the field's UTF-8 bytes, in this order; the result is hashed with SHA-256 and rendered as lowercase hex:

user id, document id, token, replacement, start position, document hash, timestamp,
previous hash, policy name, policy version, policy content hash, effective configuration hash, filename, PII type

An unset field is written as the four bytes null, and the timestamp as yyyy-MM-dd'T'HH:mm:ss.SSS'Z' in UTC. The length prefix is what fixes the boundary between one field and the next: joined with a separator alone, a token of a:b with a replacement of c and a token of a with a replacement of b:c would produce the same digest, and either could be rewritten as the other without breaking the chain.

The length is counted in bytes, not characters, because those differ for anything outside ASCII and a name with an accent is ordinary in this data. café is 5 bytes but 4 characters, and a🙂b is 6 bytes, 3 code points and 4 UTF-16 units, so a verifier counting characters would compute a different digest from Philter for exactly the values a redaction ledger is most likely to hold.

One field is not in the export: the owning account's internal user id, which is part of the hash but is not a value the export carries. An exported chain can therefore be checked for linkage (each entry's previousHash matching the hash before it, and each signature verifying against the enclosed public key), but recomputing the hashes themselves outside Philter also needs that id.

The signature proves origin. Every entry is signed with the deployment's ES256 key, so an entry rewritten in the database cannot be re-signed without that key. Signing is always on and is not tied to the output signing setting.

How strong that is depends on where the key lives. By default Philter generates the signing key and stores it in Philter's database, encrypted under PHILTER_ENCRYPTION_KEY. Database access alone is therefore not enough to forge a chain: an attacker would need the database and the deployment's master key, which is supplied from the environment rather than stored in the database.

For the strongest property, supply the signing key from outside the database entirely with PHILTER_SIGNING_KEY_PATH (see Output Signing) and restrict who can read that file. The key then never enters the database, so no amount of database access yields it.

The validity response reports both, plus how many entries carry a signature:

{
  "documentId": "...",
  "valid": true,
  "hashChainValid": true,
  "signaturesValid": true,
  "signedEntries": 2,
  "unsignedEntries": 0
}

unsignedEntries counts entries with a missing or blank signature. Any such entry makes signaturesValid and valid false, even if hashChainValid is true. A missing, unknown, or invalid signing key ID also prevents signature validation. There is no unsigned compatibility mode.

A chain that cannot be checked at all, such as one with an entry that no longer decrypts, is reported with valid false and a validationError instead of these fields. See A chain that cannot be validated.

Every new ledger entry must be signed before it is saved. If signing fails, Philter fails the redaction rather than returning success with unsigned evidence. Earlier signed entries from that request may already have been saved; this does not make the whole chain write atomic.

Verifying independently. Each entry names the key that signed it in signingKeyId. Fetch that key from GET /api/signing-key/{keyId}, which needs no authentication, then verify the ES256 signature over the entry's hash. A ledger export embeds the keys it needs in a signingKeys map, so an exported chain verifies without reaching the instance that produced it.

Key rotation preserves history. Regenerating the signing key retains the superseded key rather than deleting it, so entries signed with it stay verifiable. Without this, a single regeneration would void the provenance of every entry ever written.

What the ledger does not prove. It does not prove that a redaction was correct, only what Philter recorded. It also does not prevent deletion: an administrator with LEDGER_DELETION_ENABLED can remove entries, and a legal hold is what blocks that.

Which Policy Version Governed a Redaction

Each ledger entry records the governing policy: its name, its version (the policy's revision at the time), and a SHA-256 fingerprint of the exact policy content that was applied. These three values are part of the entry's tamper-evident hash, so the stamped policy cannot be altered without breaking the chain.

To make that version stamp resolvable to real content, Philter retains an immutable, append-only snapshot of a policy's content every time the policy is saved. Snapshots are content-addressed by their fingerprint, so:

  • a deleted-then-recreated policy that reuses a name never collides with prior evidence, and
  • editing or deleting the live policy never removes the retained snapshots. Retained versions are evidence, separate from the live policy lifecycle; removing them is a separate, deliberate, audited action.

This lets an administrator take any redaction in a date range and produce the exact policy version that was in force for it, with no engineering work, and verify the ledger chain is intact. Even if a snapshot is later removed under a retention action, the ledger entry still proves which policy was applied by its name, version, and fingerprint; only the ability to render that policy's content is lost.

The applied policy name and version are also returned at redaction time, on the X-Philter-Policy-Name and X-Philter-Policy-Version response headers of /api/filter and in the /api/explain response body. See the Redaction API.

Enabling Redaction Ledgers

Redaction ledgers are controlled on a per-context basis. Pass "ledger": true in the body when creating a context (POST /api/contexts) or updating one (PATCH /api/contexts/{name}), to turn the ledger on for that context. It is false by default, so a new context does not record a ledger until you enable it. Redactions performed in a context with the ledger enabled are recorded; redactions in a context with it disabled are not.

How and When Ledger Entries Are Deleted

Ledger entries never expire on their own. Because the ledger is a tamper-evident audit record, nothing removes an entry except a deliberate deletion by an administrator. There are two, described below.

Deletion always operates on whole document chains, never on individual entries within a chain. This preserves verifiability: a chain that remains is always complete and can still be validated, and a chain that is removed is removed in its entirety.

Legal holds block both deletion paths. If a legal hold is active on a document chain or a user's evidence, a purge or a single-chain delete against that evidence is blocked and returns HTTP 423. The hold must be released before either can proceed. Because these are the only ways entries are removed through Philter, a hold preserves the evidence it covers for as long as it is active. Securing the underlying MongoDB is a separate responsibility. See Legal Holds for the full documentation.

Deletion is restricted. Both paths require an administrator and LEDGER_DELETION_ENABLED=true, which is false by default. A deployment that has not opted in cannot delete ledger evidence through Philter at all. See Settings.

1. Purge by age (on demand or scheduled)

An administrator can prune old entries at any time. This is how you enforce a retention policy: schedule this call and you get time-based retention that is still admin-only, hold-aware, and audited.

DELETE /api/ledger?older_than_days={n} deletes the calling user's chains older than n days. See the Ledger API.

Age retention applies to a whole chain. Philter seals the chain after its ledger entries have been recorded, and further appends to that document ID are rejected. Both the completion time and the newest entry timestamp must precede the cutoff. A chain spanning the cutoff is retained in full. Open, interrupted, and failed chains are retained for review or explicit deletion. With older_than_days=0, completed chains are eligible immediately; unfinished chains remain.

The ledger_chains records keep lifecycle metadata, including a small marker after purge so a late writer cannot recreate the chain. Purge deletes entries by owner and document ID. If a database deletion fails partway through, the completion marker remains eligible for a retry; a purge across many chains is not one transaction. An interrupted operation also retains its owner guard; follow the legal-hold recovery procedure before retrying.

2. Deleting a single document's chain

DELETE /api/ledger/{documentId} removes that document's chain. See the Ledger API.

There is no automatic expiry

Earlier builds offered a REDACTION_LEDGER_TTL_DAYS variable that had MongoDB expire old entries. It was removed. Because the database performed that deletion itself, it could not check legal holds and left no audit record, which is the opposite of what evidence retention requires. Schedule the purge above instead. Startup rejects any ledger TTL index; it does not drop indexes or migrate data.

Ledger entries survive user deactivation

Deactivating a user account does not delete that user's ledger. Users are deactivated rather than deleted (see the Users API), and deactivation never cascades to the ledger: every chain is retained and stays resolvable to the retained (deactivated) owning user, so the redaction evidence is preserved. The only ways ledger entries are removed are the two above.

Exporting Ledger Entries

A document's full ledger chain can be exported as a portable JSON document so it can be archived externally and later re-verified. Each exported entry carries its hash and previousHash, so the chain's integrity can be checked offline.

GET /api/ledger/{documentId}/export returns the chain as a downloadable JSON document. See the Ledger API.

Security: unlike a context export (which contains only token hashes), a ledger export includes the decrypted original token and its replacement, because the ledger's purpose is to record exactly what was redacted to what. Both values are also inputs to each entry's hash, so they are what makes the chain verifiable offline; an export stripped of them could not be re-verified. Treat an export as sensitive and store and transmit it securely. Producing one requires an API key with the ledger:export scope.

Viewing Ledgers

Ledgers are read with the Ledger API:

  • GET /api/ledger lists your chains, most recent first. Each item is the chain's genesis entry, which carries the document ID, the original filename, and the time the chain was started. Use q to filter by all or part of a filename or document ID, and offset and limit to page.
  • GET /api/ledger/{documentId} returns a document's entries in order, a page at a time, with how many there are and whether the whole chain currently verifies. Each entry records its timestamp, its hash and the previous entry's hash, the replacement applied, and the type of information identified (for example, SSN or PERSON). The original values are returned only by an export.
  • GET /api/ledger/{documentId}/valid checks whether the chain still verifies.

An administrator can list every user's chains with GET /api/ledger?all_users=true, which requires ADMIN_CROSS_USER_ACCESS_ENABLED=true.

A chain whose head entry cannot be read (for example, it no longer decrypts after a key change) is still listed, with a readError and no replacement. Exporting a chain with an unreadable entry returns 422 with the reason entry_unreadable; nothing is exported. See the Ledger API.

Important Considerations and Limitations

  • Plain text: individual redactions are recorded as chain entries, so the ledger holds the full detail of what was redacted.
  • PDF: a chain is created and appears in the ledger listing, stamped with the filename and the governing policy version, but the individual redactions within the PDF are not recorded as entries. The chain validates, but it holds no per-redaction detail.
  • Retention: Entries are retained until you remove them (see How and When Ledger Entries Are Deleted).
  • Data Privacy: Ledgers record the original sensitive information, so treat access to them as access to the underlying data. Reading and validating a chain returns the replacements but not the original values; the originals come only from an export, which needs the separate ledger:export scope. Within that, the API key is the credential and no additional role is required for an account's own ledger. Reaching another user's ledger requires an administrator and ADMIN_CROSS_USER_ACCESS_ENABLED=true, supplied through the owner parameter on the Ledger API; every such access is audited. Scope and rotate API keys accordingly, and see Deletion is restricted for the separate, administrator-only deletion path.

Effective configuration evidence

Async ledger entries include effectiveHash, the SHA-256 of the resolved configuration captured at submission. It is included in every entry hash after the raw policy content hash. The configuration itself is stored only with the job and is removed with it, so after that the hash can be compared but not resolved; the entry's policyName, policyVersion, and policyContentHash still identify the policy revision, which policy versions keep. Synchronous entries without a captured configuration use the normal unset-field encoding (4:null). The captured configuration freezes policy dependencies and context flags; context mappings, vector contents, and software remain live. It does not guarantee byte-identical replay.

Manual document-chain deletion requires a completed chain. Open, writing, or failed publication returns 409 Conflict and retains the evidence for completion or recovery. A deletion marker prevents later appends or reuse of the deleted document ID. Internal owner-wide deletion operates on a fixed set of completed chains; chains created afterward are retained.