Skip to content

Independent R&D project · Cologne

AnyLAI

Cloud-Native and Kubernetes

Deployment boundaries, admission, network isolation, configuration provenance and release controls, with validated reference examples.

Non-normative

Companion version
1.0
Maps to COADF Core
2.2
Status
Current
Last reviewed
Profile version
1.0
Code examples
Illustrative reference examples

On this page

  1. Architectural properties addressed
  2. Architectural intent
  3. Technology mapping
  4. Reference pattern
  5. Failure modes
  6. Verification
  7. Alternative realizations
  8. Trade-offs
  9. Limitations
  10. What this profile does not establish
  11. Tested reference environment
  12. Sources

Architectural properties addressed

  • P-1The probabilistic workload in its own namespace, with network egress only to what it needs and no credential for the authoritative store. An addition to the application boundary, never a replacement.
  • P-4Execution context configured once per workload; a sampled trace pipeline kept apart from the audit trail; configuration provenance readable off every pod.
  • P-6Deployment admission and release gates as fence surfaces, with their characteristic bypasses named.
  • P-8Configuration and policy revisions that survive rollouts, and a decision record that names the revision when two are live at once.
  • P-7External services reached through explicit egress, so a new dependency is a reviewed change to policy, not a line of code.
  • P-2, P-3Not addressed at the infrastructure level: they are application properties, and COADF publishes them as principles only.

Architectural intent

Kubernetes brings boundaries of its own (namespaces, network policy, admission, rollout) and failure modes of its own: objects that are accepted and not enforced, reconciliation that happens eventually, two versions of the same service answering at once. This profile maps the COADF properties onto those boundaries without treating Kubernetes as a requirement of COADF, and without letting a cluster control stand in for an application control.

The rule that runs through it: infrastructure can make a boundary harder to cross, and it cannot make one exist. An application that writes model output into its own authoritative tables is not repaired by any network policy.

Technology mapping

Cloud-Native and Kubernetes: Technology mapping
Architectural propertyCloud-Native and Kubernetes
Probabilistic boundaryA separate Deployment and namespace; NetworkPolicy egress; the writer credential created only where the domain runs; database grants per workload
Admission fenceValidatingAdmissionPolicy (CEL, in-process) or OPA Gatekeeper (Rego, with an audit of existing resources)
Execution contextOpenTelemetry SDK configuration per workload; a Collector pipeline for traces
Audit trailThe application's own append-only store, outside the telemetry pipeline
Configuration provenanceRevision annotations, immutable ConfigMaps named by revision, images pinned by digest; a GitOps controller recording the synced revision
Release controlRequired checks that cannot pass by being skipped; deployment environments; promotion by digest
PolicyOPA as a service or sidecar, its bundle revision in decision logs; admission policies for infrastructure rules

Reference pattern

Three namespaces: inference for the model client and its boundary, model-serving for the model endpoint, records for the proposal intake, the domain core and the database. The Probabilistic Boundary page draws the same system as a cluster diagram.

P-1 · No route to the records

Illustrative reference exampleNo route from the probabilistic side to the records
Purpose
When the probabilistic side runs as its own workload, give it no route to the authoritative store.
Architectural property
Network isolation as an additional, infrastructure-level enforcement of the P-1 boundary. It adds to the application boundary and does not replace it.
networkpolicy.yamlyamlP-1
# Illustrative reference example: the inference workloads can reach the model# endpoint, the domain's proposal intake and DNS, and nothing else. The records# database sits in the same namespace as the intake, and no rule here opens a# route to it: selecting the intake pods and port is what keeps it closed.# Enforced only by a network plugin that implements NetworkPolicy, and only as an# addition: the application boundary is still required, and this replaces none of it.apiVersion: networking.k8s.io/v1kind: NetworkPolicymetadata:  name: inference-egress  namespace: inferencespec:  podSelector: {}  # every pod in this namespace  policyTypes:    - Egress  egress:    - to:        - namespaceSelector:            matchLabels:              kubernetes.io/metadata.name: model-serving      ports:        - protocol: TCP          port: 8080    - to:        - namespaceSelector:            matchLabels:              kubernetes.io/metadata.name: records          podSelector:  # same list item: this namespace AND these pods            matchLabels:              app: proposal-intake      ports:        - protocol: TCP          port: 8443    - to:        - namespaceSelector:            matchLabels:              kubernetes.io/metadata.name: kube-system          podSelector:            matchLabels:              k8s-app: kube-dns      ports:        - protocol: UDP          port: 53        - protocol: TCP          port: 53

What it intentionally omits

  • The network plugin. Without one that implements NetworkPolicy, this object is accepted and does nothing.
  • Ingress rules, and a default-deny policy for the namespace.
  • The labels of the DNS pods, which vary between clusters.
  • Egress to a model hosted outside the cluster, which needs a rule of its own.

How to verify

Not run for this Companion: the manifest was validated against the Kubernetes schemas, which proves its shape and nothing about enforcement. To verify enforcement, in a test cluster with an enforcing network plugin, from an inference pod: the model endpoint answers and the records database does not; delete the policy and the database answers, which proves it was the policy that blocked it.

Failure mode addressed

A NetworkPolicy committed to a cluster whose network plugin does not enforce it. The manifest is valid, the review passes, and nothing is isolated.

Relies on

Kubernetes: Network Policies: prerequisites

P-6 · A deployment-time fence

ValidatingAdmissionPolicy is stable since Kubernetes 1.30 and runs inside the API server, so it needs no webhook to be available. The same rules can be written for OPA Gatekeeper, which adds an audit of resources that already exist and enforcement actions such as dryrun and warn; both are valid, and the property does not depend on the choice.

The rule has to check the whole thing it names. A test for @sha256: somewhere in the image reference admits image@sha256: with nothing after it; this policy matches a complete SHA-256 digest, and requires the revision annotation to hold a revision, not merely to exist.

Illustrative reference exampleA deployment-time fence for configuration provenance
Purpose
Admit no Deployment whose images are not pinned by a well-formed digest, or that cannot name, in a non-empty annotation, the configuration revision it came from.
Architectural property
A forbidden state prevents the transition, here the creation or update of a Deployment (P-6), and configuration provenance becomes a property of every running workload (P-4).
admission-policy.yamlyamlP-4 · P-6
# Illustrative reference example: a Deployment is admitted only if every image# reference ends in a well-formed sha256 digest and the object names, in a# non-empty annotation, the configuration revision it was rendered from.# Admission sees API objects; it never sees application data.apiVersion: admissionregistration.k8s.io/v1kind: ValidatingAdmissionPolicymetadata:  name: deployments-declare-provenancespec:  failurePolicy: Fail  matchConstraints:    resourceRules:      - apiGroups: ["apps"]        apiVersions: ["v1"]        operations: ["CREATE", "UPDATE"]        resources: ["deployments"]  validations:    - expression: >-        object.spec.template.spec.containers.all(c,          c.image.matches('^[^@]+@sha256:[0-9a-f]{64}$')) &&        (!has(object.spec.template.spec.initContainers) ||          object.spec.template.spec.initContainers.all(c,            c.image.matches('^[^@]+@sha256:[0-9a-f]{64}$')))      message: "every container image must end in a sha256 digest of 64 hex characters"    - expression: >-        has(object.metadata.annotations) &&        'refapp.example/config-revision' in object.metadata.annotations &&        object.metadata.annotations['refapp.example/config-revision'].matches('^[A-Za-z0-9._-]+$')      message: "the Deployment must name, in a non-empty annotation, the configuration revision it was rendered from"---apiVersion: admissionregistration.k8s.io/v1kind: ValidatingAdmissionPolicyBindingmetadata:  name: deployments-declare-provenancespec:  policyName: deployments-declare-provenance  validationActions: ["Deny"]  matchResources:    namespaceSelector:      matchLabels:        refapp.example/provenance-required: "true"

What it intentionally omits

  • Application-level validation. Admission sees API objects and never the requests your service handles.
  • Objects that already exist. Admission evaluates requests as they are made, and adding this policy re-checks nothing that is already running.
  • Pods created directly, and other workload kinds (StatefulSet, Job, CronJob, DaemonSet), which each need their own match.
  • Whether the named revision exists, or whether the digest is the one continuous integration built. The rule checks the form of a digest, not its origin.

How to verify

Against a real Kubernetes API server (a disposable local kube-apiserver 1.37 with no nodes; every case a server-side dry run): images pinned by digest are admitted, including behind a registry port and after a tag; a tag-only image, an empty digest, a digest that is not hex and a digest of the wrong length are refused, and so is a missing, empty or blank revision annotation (tests/test_admission_policy.py). With a presence-only rule, contains('@sha256:') and a bare in, five of those cases are admitted, which is why the rule matches the whole digest.

Failure mode addressed

Unverified release assumptions: a tag moved after review, or a running Deployment that nobody can map back to the configuration it was rendered from.

Relies on

Kubernetes: Validating Admission Policy · Kubernetes: Admission control: what are they · Kubernetes: Limit Ranges: admission-time validation · Kubernetes: Images: image names and digests · Open Container Initiative: Image specification, descriptors: SHA-256 digests

P-4 and P-8 · Configuration provenance at the pod

Illustrative reference exampleConfiguration provenance you can read off a running pod
Purpose
Make the configuration revision visible in the object, the pod template and the name of an immutable ConfigMap.
Architectural property
A new revision is a rollout, and no pod runs a configuration it cannot name. The prompt template revision the model call reports (prompt_revision in the Python boundary) has a deployable counterpart here.
deployment.yamlyamlP-4 · P-8
# Illustrative reference example: configuration provenance you can read off a# running pod. The revision is in the object, in the pod template and in the# name of the immutable ConfigMap, so a new revision is a new rollout and no# pod runs a configuration it cannot name.apiVersion: v1kind: ConfigMapmetadata:  name: prompt-templates-3f9c2d1  namespace: inferenceimmutable: truedata:  service-report.txt: |    Read the service report and propose next_service_due and component_class.---apiVersion: apps/v1kind: Deploymentmetadata:  name: report-extraction  namespace: inference  annotations:    refapp.example/config-revision: "3f9c2d1"spec:  replicas: 2  selector:    matchLabels:      app: report-extraction  template:    metadata:      labels:        app: report-extraction      annotations:        refapp.example/config-revision: "3f9c2d1"    spec:      automountServiceAccountToken: false      containers:        - name: worker          image: registry.example/refapp/extraction@sha256:3f9c2d1e3f9c2d1e3f9c2d1e3f9c2d1e3f9c2d1e3f9c2d1e3f9c2d1e3f9c2d1e          env:            - name: OTEL_SERVICE_NAME              value: report-extraction            - name: OTEL_RESOURCE_ATTRIBUTES              value: service.version=3f9c2d1          volumeMounts:            - name: prompts              mountPath: /etc/refapp/prompts              readOnly: true      volumes:        - name: prompts          configMap:            name: prompt-templates-3f9c2d1

What it intentionally omits

  • Rollout overlap. During a rolling update old and new pods serve at the same time, so two revisions are live at once; the decision record, not the deployment state, is what must name the revision.
  • Secrets, probes, resources and security context.
  • The service account and its permissions.

How to verify

Edit the prompt text under the same name: the ConfigMap is immutable and the change is refused. Create a new ConfigMap and update the name and the annotations: a rollout follows, and each pod reports its revision as service.version.

Failure mode addressed

Environments running different configuration without provenance. A ConfigMap read through environment variables is not picked up until pods restart, a mounted one is picked up eventually, and in between nobody can say which configuration produced a given answer.

Relies on

Kubernetes: ConfigMaps: updates and immutability · Kubernetes: Deployments: rolling update · Kubernetes: Annotations

P-4 · The trace pipeline is not the audit trail

OpenTelemetry SDKs read OTEL_PROPAGATORS and default to tracecontext,baggage, so the execution context crosses service boundaries without code. The audit identity does not travel that way: it is part of each message's contract, and the audit entries are written by the application to its own store.

Illustrative reference exampleA trace pipeline that samples, and is therefore not the audit trail
Purpose
Show a normal, sampled trace pipeline, and why the audit trail never goes through one.
Architectural property
Execution telemetry and the audit trail are different artifacts, and they promise different things. They correlate through the audit trace_id, and the audit trail never passes through a pipeline configured to drop data (P-4).
otel-collector.yamlyamlP-4
# Illustrative reference example: an OpenTelemetry Collector pipeline for# traces. Sampling is legitimate here, and it is exactly why this pipeline is# not the audit trail: audit entries are written to their own append-only# store by the application, and never pass through this configuration.receivers:  otlp:    protocols:      grpc: {}      http: {} processors:  memory_limiter:    check_interval: 1s    limit_percentage: ${env:MEMORY_LIMIT_PERCENTAGE}  probabilistic_sampler:    sampling_percentage: ${env:TRACE_SAMPLE_PERCENTAGE}  batch: {} exporters:  otlp:    endpoint: tracing-backend.observability:4317 service:  pipelines:    traces:      receivers: [otlp]      processors: [memory_limiter, probabilistic_sampler, batch]      exporters: [otlp]

What it intentionally omits

  • Tail sampling, which needs every span of a trace to reach the same Collector instance.
  • The trace backend and its retention.
  • Log and metric pipelines.
  • The percentages. They are operational settings, supplied through environment variable expansion, and their values are irrelevant to the architectural property.

How to verify

otelcol-contrib validate accepts the configuration with the two percentages supplied by the environment, and refuses it when they are missing. Send a known number of traces and count what arrives: with any sampling that drops traces, some do not arrive, which is the property that disqualifies this path as an audit trail.

Failure mode addressed

A trace present only in the tracing backend: the question 'what happened to this transaction' is answered from a store that is allowed to have dropped it.

Relies on

OpenTelemetry: Sampling · OpenTelemetry: Tracing SDK: sampling · OpenTelemetry: Transforming telemetry · OpenTelemetry Collector Contrib: Tail sampling processor · OpenTelemetry: Collector configuration: environment variables

P-6 · Release controls

The workflow defines the checks and fails its summary job when a fence fails, is cancelled or is skipped. It cannot make that job required: that is branch protection or a ruleset in the repository settings, and without it the merge button ignores the result.

Illustrative reference exampleFences that stop a release, and a check that cannot pass by being skipped
Purpose
Stop the merge and the release on a failed fence, with one required check that stays red when a fence is skipped.
Architectural property
A failed, cancelled or skipped fence fails the fences job, and the release does not run (P-6). The workflow cannot make fences required: only branch protection or a ruleset in the repository settings blocks a merge on it.
release-workflow.yamlyamlP-6
# Illustrative reference example: fences that stop a release, and one summary# check that cannot turn green by being skipped. Making it a REQUIRED check is a# repository setting (branch protection or a ruleset), not part of this file.name: releaseon:  pull_request:  push:    branches: [main] jobs:  architecture:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v7      - uses: actions/setup-python@v7        with:          python-version: "3.12"      - run: pip install -r requirements.txt      - run: lint-imports   tests:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v7      - uses: actions/setup-python@v7        with:          python-version: "3.12"      - run: pip install -r requirements.txt      - run: pytest -q   publication:    runs-on: ubuntu-latest    steps:      - uses: actions/checkout@v7      - run: make site  # writes the public build to build/      - name: Fence the built output        env:          PATTERNS: ${{ secrets.PUBLICATION_PATTERNS }}        run: |          printf '%s\n' "$PATTERNS" > "$RUNNER_TEMP/patterns.txt"          python -m refapp.fence build/ "$RUNNER_TEMP/patterns.txt"   # The one job branch protection requires. A skipped job reports success,  # so this job runs whatever happened above and fails unless all passed.  fences:    if: ${{ always() }}    needs: [architecture, tests, publication]    runs-on: ubuntu-latest    steps:      - if: >-          contains(needs.*.result, 'failure') ||          contains(needs.*.result, 'cancelled') ||          contains(needs.*.result, 'skipped')        run: exit 1   release:    if: github.event_name == 'push'    needs: fences  # never `if: always()` here: that would release after a failed fence    runs-on: ubuntu-latest    environment: production    steps:      - run: echo "promote the artifact built from ${{ github.sha }}"

What it intentionally omits

  • The site build itself; make site stands in for it.
  • Branch protection settings, which live in the repository settings and not in the workflow.
  • Pinning actions to commit hashes, the stronger practice. Major-version tags are used here for readability.

How to verify

Make one fence job fail on a test branch: the fences job fails and the release is skipped. Make a fence job skip instead, with a false if:: the fences job still fails, because a skipped job reports success to branch protection and this job refuses to count it.

Failure mode addressed

A required check that turns green by being skipped. GitHub reports a skipped job as success, so a required job that depends on a failed fence can let the merge through.

Relies on

GitHub: Workflow syntax: jobs.<job_id>.needs · GitHub: Control jobs with conditions · GitHub: Troubleshooting required status checks · GitHub: Deployments and environments: required reviewers · GitHub: About protected branches: require status checks before merging

Failure modes

  1. A NetworkPolicy nothing enforces

    A NetworkPolicy without a network plugin that implements it has no effect. The manifest is valid, the review passes, and nothing is isolated.

  2. One hyphen widens the rule

    Inside one to entry, a namespaceSelector and a podSelector together select those pods in those namespaces. Written as two list items, they select the whole namespace, or those pods in the policy's own namespace. The Kubernetes documentation warns about exactly this YAML difference.

  3. Admission mistaken for application validation

    Admission intercepts requests to the API server: it sees Deployments and Pods, never the requests your service handles or the replies a model returns. See the technology note.

  4. A fence that fails open

    A webhook whose failure policy is Ignore lets a request through when the webhook cannot be called. The day the webhook is down is the day the rule is off.

  5. Existing objects never rechecked

    Admission judges requests as they arrive. For LimitRange, Kubernetes states that pods already running continue unchanged; a new admission rule likewise says nothing about what was admitted before it. Gatekeeper's audit exists for this gap.

  6. A tag moved after review

    Digests are immutable; tags can be moved to point to different images. An image reviewed by tag is not necessarily the image running.

  7. Configuration drift you cannot see

    A ConfigMap consumed through environment variables is not updated until the pod restarts, a mounted one is updated eventually, and one mounted through subPath is never updated. Three pods can hold three configurations while every manifest names one.

  8. Two revisions live during a rollout

    Rolling updates run old and new versions at the same time. Anything that has to name a revision (a decision, an extraction, an answer) must record it at the moment it is produced, not infer it from the deployment afterwards.

  9. The sampled pipeline used as a record

    A trace that is not sampled is not exported. A pipeline configured to keep a fraction of traces is correct for diagnosis and cannot answer an audit question.

  10. Baggage that leaves the building

    OpenTelemetry warns that baggage can be propagated to unintended recipients, third-party APIs included. An identifier put in baggage for convenience travels further than its owner intended.

  11. A synced revision read as the deployed revision

    Argo CD's status.sync.revision is the revision the live state was last compared against; the sync history records the revision each sync actually applied. They are the same only while the application is in sync.

  12. A named-person gate the plan does not enforce

    On GitHub Free, Pro and Team plans, required reviewers on environments are available only for public repositories. A private repository on those plans has the setting's name and not its effect.

Verification

  • Deployment or admission test

    Passes when: Against a disposable local API server, the example Deployment is admitted into a namespace carrying the binding's label, and so are digests behind a registry port or after a tag.

    Proof of teeth: A tag-only image, an empty or malformed digest, and a missing, empty or blank revision are each refused by the policy. Replace the digest and revision rules with presence checks: five of those cases are admitted, and the tests fail.

  • Integration test

    Passes when: Not run for this revision. From an inference pod, with an enforcing network plugin: the model endpoint and the proposal intake answer, and the database port does not.

    Proof of teeth: Delete the policy: the database answers, which proves that the policy, and not something else, was blocking it.

  • Contract test

    Passes when: Manifests validate against the Kubernetes schemas of every supported minor version, in strict mode, so a misspelt field is an error.

    Proof of teeth: Misspell podSelector: strict validation fails.

  • Contract test

    Passes when: The Collector configuration validates with the Collector binary of the version deployed, with the percentages supplied by the environment.

    Proof of teeth: Misspell a processor setting, or leave the percentages unset: validation fails.

  • End-to-end test

    Passes when: A failed fence job fails the summary job and skips the release; branch protection lists the summary job as required.

    Proof of teeth: Skip a fence job instead of failing it: the summary job still fails, because it refuses to count a skipped job.

  • Manual evidence

    Passes when: At each release, the image digests and the configuration revision are recorded, and compared with what the cluster reports.

Alternative realizations

  • Gatekeeper, Kyverno or ValidatingAdmissionPolicy for admission rules. The property is that the rule stops the transition and has been watched doing so; the engine is a choice.
  • A service mesh's authorisation policy where egress needs to be controlled per request rather than per port.
  • GitOps controllers (Argo CD, Flux) as the record of which configuration revision is applied, in addition to the annotations on the objects.
  • A managed platform without Kubernetes. Every property in this profile has an equivalent there: separate deployables, network rules, deployment gates, configuration revisions. COADF requires none of the tools named here.

Trade-offs

  • In-process admission versus a policy engine. ValidatingAdmissionPolicy needs no extra component and cannot be down separately from the API server; Gatekeeper adds an audit of existing resources and staged enforcement, and is one more system to run.
  • Network isolation has a price: an enforcing plugin, policies to maintain, and failures that look like application bugs when a rule is too tight.
  • Separate deployments turn one boundary into a versioned contract that has to stay compatible across rollouts, in both directions.
  • Sampling saves cost and removes evidence value. Keep it, and keep the audit trail elsewhere.

Limitations

  • The manifests were validated against the Kubernetes schemas for 1.35, 1.36 and 1.37; the admission policy was applied to a disposable local API server with no nodes and exercised by server-side dry runs; the Collector configuration was validated and the workflow was linted. Nothing ran on a cluster with nodes: the network policy's enforcement, rollouts and the Collector's runtime behaviour were not observed.
  • The examples omit ingress rules, security contexts, resources, probes and secrets.
  • Kubernetes is one way to deploy these properties. COADF does not require it.

What this profile does not establish

Following this profile does not establish regulatory compliance, certification or conformity assessment, and nothing here is required by COADF. A cluster that admits these manifests has admitted these manifests.

Tested reference environment

  • kubeconform 0.8.0 · strict schema validation against Kubernetes 1.35.0, 1.36.0 and 1.37.0; not applied to a cluster
  • Kubernetes API server (envtest release) 1.37.0 · a disposable local kube-apiserver with etcd and no nodes: the admission policy applied and ten cases run as server-side dry runs; nothing scheduled, no network enforcement
  • OpenTelemetry Collector (contrib) 0.160.0 · otelcol-contrib validate
  • actionlint 1.7.12 · without shellcheck, so the shell inside run steps was not linted

Sources

COADF Engineering Companion 1.0 · non-normative · maps to COADF Core 2.2

Publication rights reserved. No public licence is granted for the COADF Engineering Companion 1.0 or its reference examples at this time.

IP and publication status