Kubernetes: enforcement (experimental, opt-in)¶

Enforcement is an optional, default-off path layered on top of the observe foundation. It guards a static, administrator-assigned tenant node pool with three cooperating pieces: a controlled kube-scheduler plugin, a fail-closed admission webhook, and tenant RBAC.

Simulated software integration — not real GPU isolation

The lab demo proves that a bypass is blocked and that a bind is or is not admitted. It does not prove real GPU isolation, attestation, or that a specific physical GPU was prepared. mock evidence is a software-integration stand-in only. The production profile rejects mock (SOURCE_NOT_ACCEPTED); the lab profile accepts it strictly as simulated integration. Enable with enforcement.enabled=true.

One evaluator, fail-closed everywhere¶

The scheduler plugin calls xila-guard-core (POST /v1alpha1/evaluate) for every placement decision — no policy semantics live in Go. A protected placement is admitted only on an explicit core allow; every other condition blocks.

Condition	Filter result	PreBind result
node not in protected pool	Success	Success
core `allow`	Success	Success
core `deny`	`Unschedulable`	rejected (requeue)
`indeterminate` / incomplete / stale / source-not-accepted	`UnschedulableAndUnresolvable`	rejected (requeue)
evaluator timeout / unreachable / non-2xx	`UnschedulableAndUnresolvable`	rejected (requeue)
pod tenant ≠ pool tenant	`UnschedulableAndUnresolvable`	rejected (requeue)
identity unresolved / no policy	`UnschedulableAndUnresolvable`	rejected (requeue)
node UID / incarnation changed since Filter	—	rejected (requeue)

Transport failure is never allow. The webhook (failurePolicy: Fail) blocks protected Pods when it is unavailable.

The controlled scheduler¶

cmd/scheduler is a kube-scheduler binary carrying the XilaGuardEnforcement framework plugin, pinned to Kubernetes 1.35 (k8s.io/kubernetes v1.35.3). Its stages:

flowchart LR
    PF[PreFilter<br/>snapshot tenant + policy +<br/>protected NodeSecurityStates once/cycle] --> F[Filter<br/>admit only core allow<br/>fail-closed otherwise]
    F --> PB[PreBind<br/>re-read fresh evidence;<br/>reject on not-allow /<br/>node-UID or incarnation change / stale]

Profile name: xila-guard-scheduler. Pool label: guard.xila.us/tenant-pool.
Env: XILAGUARD_CORE_URL (default http://127.0.0.1:8080), XILAGUARD_ENFORCE_ACCEPT_MOCK (default false → production profile rejects mock), XILAGUARD_EVAL_TIMEOUT_MS (default 800), XILAGUARD_POOL_LABEL.
Remote attestation and slow hardware discovery are kept out of per-node Filter calls; the plugin uses a cached per-cycle snapshot.

bin/scheduler --config config/scheduler/config.yaml --kubeconfig "$KUBECONFIG"

The fail-closed admission webhook¶

cmd/webhook serves AdmissionReview v1 on /validate-pods. For protected Pods — including Pods that omit GPU requests — it rejects:

an alternate schedulerName;
a direct spec.nodeName binding;
privileged / hostNetwork / hostPID / hostIPC / hostPath;
an unresolved tenant.

Trusted infrastructure DaemonSet service accounts are an explicit, documented exemption (--trusted-serviceaccounts). The paired ValidatingWebhookConfiguration uses failurePolicy: Fail, so an unavailable webhook blocks rather than silently admits.

bin/webhook --listen :8443 --tls-cert-file tls.crt --tls-key-file tls.key \
  --core-url http://127.0.0.1:8080 --trusted-serviceaccounts kube-system/node-exporter

Tenant RBAC¶

Tenants may manage their own Pods but cannot read or modify PlacementPolicy, TenantBinding, or NodeSecurityState, cannot create pods/binding, and cannot patch nodes. A selector a tenant can remove is not a boundary — admission, scheduler filtering, and RBAC must work together.

Protected pool model¶

Nodes in the pool carry the admin-set label guard.xila.us/tenant-pool=<tenantId>. The label is immutable while tenant workloads are present: reassigning a pool is an explicit maintenance procedure (drain, then relabel) outside this prototype.
The controlled route is schedulerName: xila-guard-scheduler. The authoritative TenantBinding.effectivePolicyRef wins; the guard.xila.us/policy annotation is a fallback.

The device-allocation limit (explicit)¶

The scheduler evaluates node-scope policy over every potentially-allocatable device on a candidate node. A mixed or incomplete pool yields indeterminate and the node is filtered out. The scheduler never claims it selected a particular physical GPU. PreBind re-reads fresh policy / evidence and re-evaluates, which narrows the Filter→bind window, but cannot prove state remains unchanged until execution. Stronger device-specific guarantees require an authoritative allocation hook and a launch / key-release boundary — out of scope here.

A live run¶

From make demo-m3 (3-node kind v0.31 / k8s 1.35), all simulated over mock:

PHASE 1 production profile  -> pod Pending: FailedScheduling ... SOURCE_NOT_ACCEPTED (mock rejected)
PHASE 2 lab-simulated       -> pod Scheduled to xila-guard-lab-worker ONLY (not the time-sliced worker2)
PHASE 3 pinned to bad node  -> pod Pending: policy deny [TIME_SLICING_FORBIDDEN]
PHASE 4 webhook             -> DENIED: alt-scheduler, nodeName, privileged
PHASE 5 RBAC                -> tenant: create pods OK; patch NSS / get policy / patch binding / binding / patch nodes DENIED

This is simulated integration over mock evidence — no physical GPU is selected. See the enforcement-boundary ADR and Helm chart for deployment.