End-to-End Traceability

On this page

Architectural property
Why it matters
Valid implementation strategies
Failure modes
Verification
Alternative realizations
Limitations
Sources

Architectural property

COADF P-4 is published in full. An identifier is created when a document is taken in and travels with every processing step, every confidence assessment, every human decision and the published output. The trail is append-only, one query reconstructs the chain, and the chain exports as JSON for somebody who did not build the system. The published audit trail schema names the fields: trace_id, timestamp, event_type, actor, input, output, decision, source_hash and immutable.

Two properties carry it. Continuity: every step carries the same identity, across every boundary it crosses. Persistence: the record of each step survives, and nobody can edit it to make the answer look better later.

Why it matters

The question P-4 answers arrives late and from outside. Months after publication somebody asks where a value came from, what read it, and who looked at it. The answer has to be reconstructable by a person who did not build the system, from records nobody could have rewritten in the meantime.

A distributed trace is not a business audit trail

This is the distinction most implementations blur, because the tools look alike: both have an identifier, both cross services, both show steps in order. They answer different questions and make different promises.

Distributed trace and audit trail, compared
Aspect	Distributed trace	Audit trail
Purpose	Explain an execution: latency, errors, call structure	Establish what happened to a transaction, and who acted
Identity	One trace per request or job, carried in the W3C `traceparent` header	One `trace_id` per business transaction, minted at intake
Lifetime	The request's duration; retention measured in days	As long as the records it justifies
Sampling	Sampling is normal and allowed	Not sampled: a missing entry is a defect
Mutability	Pipelines filter, drop and rewrite by configuration	Append-only: a correction is a new entry
Content	Spans, timings, attributes chosen for diagnosis	Business events: input, output, decision, actor

The W3C Trace Context specification defines the traceparent header, and OpenTelemetry SDKs propagate tracecontext and baggage by default. That identity is per execution. In OpenTelemetry, a trace that is not sampled is not exported, and the Collector's filter processor drops telemetry that matches a condition. Both are correct behaviour for diagnosis and disqualifying for evidence.

The two correlate: record the audit trace_id on every span as an attribute, and the distributed trace identifier in the logs. Do not make either one the other. A single business transaction routinely spans several distributed traces: an intake request, an asynchronous processing job, a decision made days later, the output.

Correlation copies the audit identifier into telemetry, and telemetry crosses other boundaries: exporters, vendors, access rules and retention that are usually looser than the audit store's. Classify the identifier before it leaves: an opaque reference, never personal data or a secret. Where the audit identifier carries meaning of its own, correlate through a separate, non-sensitive reference.

Example architecture · Non-normative

Distributed traces and one audit trail

Correlation through the audit trace_id

One business transaction, four distributed traces. Telemetry may be sampled and is short-lived; audit entries are persisted separately and are not sampled. They correlate through the audit trace_id, recorded on each trace as a span attribute. The events are illustrative, not a required COADF workflow.

Text description. Top lane, distributed traces: four separate traces, one each for an intake request, a processing job, a decision made days later, and an output. They are not connected to one another. Bottom lane, the audit trail: four illustrative entries, intake, processing, decision and output, all carrying the same trace_id, persisted separately from telemetry and not sampled. A note in the lane says the events are illustrative, not a required COADF workflow. A dashed line joins each trace to the entry it corresponds to, labelled as a span attribute carrying the audit trace_id.

Valid implementation strategies

Mint once, at intake

The identifier is created where the transaction enters and nowhere else. Every later component requires it and none generates one. A missing identifier is an error, never a reason to create a fresh one, and a default value in a message model is the most common way that rule is broken without anyone noticing.

Carry it explicitly across every boundary

In process: a function argument or the request context.
Between services: a field in the request or message contract, where a schema can require it.
In scheduled work: a job argument.
At rest: a column on every stored row.

Propagation libraries carry the execution context for you. The audit identity is part of your contract and belongs in the payload.

Persist, append-only

Several realisations are valid, and they combine:

Insert-only persistence. The repository offers append and read, and nothing else.
Database privileges. The application's role has INSERT and SELECT on the trail, and no UPDATE, DELETE or TRUNCATE.
Triggers that refuse rewrites, as defence in depth for sessions that do hold broader rights, with a statement-level trigger for TRUNCATE, because in PostgreSQL TRUNCATE does not fire ON DELETE triggers.
An immutable event model, in which a correction is a new event that supersedes an earlier one.
A superseding convention. The current value of an attribute is the latest entry for it; the history is all of them.

Event sourcing is one way to obtain an append-only history, and not the only one; see the technology note.

Write the entry with the change

Record the audit entry in the same database transaction as the state change it describes, or through a transactional outbox, so that neither can exist without the other.

Make retries idempotent

Stamp the step's time when the step runs, and resend the same entry on retry. Give entries a natural unique key so a repeated write is absorbed rather than duplicated. In PostgreSQL, ON CONFLICT DO NOTHING skips a row that conflicts with a unique constraint or index instead of raising an error, which is exactly why it cannot be the whole answer: a conflict is not always a retry. The same key with different content is a second writer or a bug. Compare the stored entry before calling the write a retry, and fail loudly when it differs.

Reconstruct with one query, export as JSON

If reconstruction needs somebody who knows the system, P-4 does not hold. Order by timestamp, and state what that order does not settle: entries with the same timestamp are not ordered by any of the published fields.

What a trigger does not give you

Privileges and triggers are controls inside the database's own trust boundary. In PostgreSQL, a superuser bypasses all permission checks, and the table's owner can disable its triggers. An append-only table is therefore a strong control against the application and a weak one against its administrators. Tamper evidence against privileged insiders needs a control outside the database, and it is a separate property from append-only persistence. Neither is, by itself, a legal statement about evidence.

Failure modes

The identifier regenerated halfway
A consumer or worker mints a fresh identifier when the field is absent, often through a default value in a message model. The result is two halves of one transaction and no query that joins them.
The trace exists only in logs
Logs rotate, are sampled, are unstructured, and are written by code that treats them as diagnostics. A chain that can only be rebuilt from logs cannot be rebuilt reliably.
An asynchronous boundary loses the context
Thread pools, executors and brokers carry nothing unless something copies it. In Python, asyncio.to_thread is documented to propagate the current context; run_in_executor documents no such thing, and CPython's own to_thread copies the context explicitly before calling it. In Spring, an executor needs a context-propagating task decorator.
Stored rows cannot be related to their evidence
No source_hash, or a hash of something other than the bytes that were actually read, so the same input can never be recognised again.
Retries produce ambiguous entries
A timestamp taken at insert time turns each retry into a new, different entry, and the trail then says a step happened twice.
A conflict discarded as a retry
A unique key and ON CONFLICT DO NOTHING, and nothing else: a second entry with the same key and a different decision disappears without an error, and the trail keeps whichever arrived first.
A correction overwrites
An UPDATE destroys what was believed before, and the trail can no longer explain why an earlier output said what it said.
A service boundary starts a new trace
A gateway strips a header it does not know, a client library does not propagate, a batch job starts from nothing. Each is invisible until somebody tries to reconstruct.
The tracing backend used as the audit trail
Its sampling, retention and mutable pipeline are right for diagnosis and wrong for evidence.
Wall clocks used to order steps across hosts
Timestamps from different machines order entries only as well as their clocks agree. Within one transaction, a single writer, or an order assigned by one, is more reliable than wall-clock time from several hosts.

Verification

Integration test
Passes when: From a published output back to its sources: one query returns every entry, all with the same trace_id, with the source_hash of each document read.
Proof of teeth: Drop the identifier from one asynchronous hop in a scratch branch. The reconstruction test must fail, not return a shorter chain that looks complete.
Integration test
Passes when: A correction appends: after it, both entries exist, and the later one supersedes the earlier.
Proof of teeth: Replace the append with an update: the test fails.
Integration test
Passes when: An exact retry is absorbed, and the same key with different content is refused, not absorbed.
Proof of teeth: Take every conflict as a retry without comparing: the test that writes different content under the same key fails.
Integration test
Passes when: Against the real database, the application's role cannot update, delete or truncate the trail.
Proof of teeth: Grant UPDATE to the role in a scratch database: the test fails.
Contract test
Passes when: Every message schema that crosses an asynchronous boundary requires the audit identifier.
Proof of teeth: Make the field optional: the contract test fails.
End-to-end test
Passes when: A request that fans out to a job and a message keeps one audit identity, checked in the persisted trail rather than in the logs.
Manual evidence
Passes when: Export one chain as JSON and have a person who did not build the system reconstruct the output's history from it alone.

Alternative realizations

Event sourcing. The event store is the history and state is derived from it. Strong audit properties, and a large architectural commitment.
Change data capture. Row changes read from the database's log. Useful for replication; it records what changed, not who decided or why, so it does not replace business audit entries.
Write-once object storage for exported chains, where retention has to outlive the database.
Ledger or tamper-evident stores, where insider tampering is part of the threat model.

Limitations

The pattern does not decide which steps are relevant for a particular regulation or a particular product.
Append-only persistence is not tamper evidence, and neither is a legal conclusion about evidence.
It does not define retention periods.
It relies on every component honouring the contract. A component that does not is found by the reconstruction test, not prevented by it.

Sources

W3C: Trace Context, the traceparent header · specification · W3C Recommendation, Level 1 · Checked on 2026-09-11
OpenTelemetry: SDK environment variables: OTEL_PROPAGATORS · specification · Specification 1.60.0 · Checked on 2026-09-11
OpenTelemetry: Sampling · official documentation · Checked on 2026-09-11
OpenTelemetry: Transforming telemetry · official documentation · Checked on 2026-09-11
PostgreSQL Global Development Group: TRUNCATE · official documentation · PostgreSQL 18 · Checked on 2026-09-11
PostgreSQL Global Development Group: INSERT: ON CONFLICT · official documentation · PostgreSQL 18 · Checked on 2026-09-11
PostgreSQL Global Development Group: Role attributes · official documentation · PostgreSQL 18 · Checked on 2026-09-11
PostgreSQL Global Development Group: ALTER TABLE: DISABLE TRIGGER · official documentation · PostgreSQL 18 · Checked on 2026-09-11
Python Software Foundation: asyncio: Task and to_thread · language documentation · Python 3.14 · Checked on 2026-09-11
CPython: Lib/asyncio/threads.py · project repository · CPython 3.14.7 · Checked on 2026-09-11
Spring Framework: Observability support: context propagation · official documentation · Spring Framework 7.0 · Checked on 2026-09-11