Orchestrator integrations

Xila Guard can coexist with third-party GPU orchestrators / schedulers — run:ai, Volcano, Kueue, YuniKorn, … — and enforce its placement-security boundary on their workloads without modifying, rebuilding, importing, or opening a PR against any of them. The integration is a co-registered Kubernetes admission webhook plus the existing observe controller. It is opt-in and default-off; enabling it is one or two operator changes.

An orchestrator is a profile, not code

Everything needed to coexist with an external orchestrator reduces to two generic data values:

• externalSchedulerName — the schedulerName the orchestrator sets on its Pods (e.g. run:ai uses runai-scheduler). A protected Pod using it is a legitimate external scheduling route, not an alternate-scheduler bypass.
• workloadSelector — a label selector that picks the orchestrator's workload Pods, used as the admission webhook's objectSelector.

An orchestrator profile is exactly that pair (internal/integration.OrchestratorProfile). Adding support for a new orchestrator is a configuration entry, not code — no vendor Go type is ever imported; internal/integration is pure data and the webhook targets plain v1.Pod.

Why NOT an in-tree scheduler plugin

run:ai's scheduler (KAI / dromos) registers its plugins at compile time. An in-process plugin would require rebuilding run:ai, which is explicitly out of bounds. A co-registered ValidatingWebhookConfiguration is the supported, vendor-neutral extension point: the orchestrator's own admission is a standard validating webhook, so adding another validating webhook on the same objects is fully supported — all registered webhooks must pass.

Two paths, and the honest granularity limit

flowchart LR
    POD[External-orchestrator Pod CREATE] --> WH[Co-registered admission webhook<br/>node-independent boundary gate]
    WH -->|fail-closed| ADMIT{admit?}
    ADMIT -->|no| BLOCK[BLOCKED]
    ADMIT -->|yes| EXT[External scheduler picks the node]
    EXT --> BOUND[Pod bound]
    BOUND --> OBS[Observe controller<br/>node-scoped policy + drift]
    OBS --> REP[report-only Events / JSONL]

Path	When	Scope	Action
Admission (co-registered webhook)	at Pod CREATE	node-independent boundary gate	BLOCKS fail-closed
Observe (controller)	post-binding	full node-scoped policy eval + drift	report-only (Events / JSONL)

For a protected Pod the webhook still enforces, fail-closed: the tenant must resolve; no direct spec.nodeName pin; no privileged / hostNetwork / hostPID / hostIPC / hostPath; trusted-DaemonSet service accounts are an explicit exemption. The only check relaxed in profile mode is the scheduler route — a Pod whose schedulerName is in the accepted set is allowed instead of being required to use xila-guard-scheduler.

The honest limit

Node-scoped blocking at schedule-time is not possible out-of-tree for ANY external scheduler without modifying it — the external scheduler picks the node and we cannot hook its decision. Node-scoped evaluation + drift is therefore done by the observe controller after binding, and is report-only. Admission here does node-independent checks only; it does not and cannot claim it picked or vetted a specific GPU / node at schedule time.

Profiles

Profile	schedulerName	workloadSelector key	Source
runai	runai-scheduler	run.ai/application (Exists)	VERIFIED (run:ai source)
volcano	volcano	volcano.sh/job-name (Exists)	VERIFIED (Volcano source + live e2e, v1.12.2)
kueue	default-scheduler	kueue.x-k8s.io/queue-name (Exists)	EXAMPLE / best-effort
yunikorn	yunikorn	applicationId (Exists)	EXAMPLE / best-effort

Verification status is explicit

run:ai is verified against the orchestrator's source. Volcano is verified both from upstream source and with a live kind end-to-end against real, unmodified Volcano v1.12.2. Kueue and YuniKorn are EXAMPLE / best-effort — confirm the scheduler name and workload label against your deployment before relying on them.

Adding a new orchestrator (no code)

orchestratorIntegrations:
  - name: my-orchestrator        # DNS-safe; names <release>-xila-guard-webhook-my-orchestrator
    schedulerName: my-scheduler  # the scheduler it sets on its pods ("" = default scheduler)
    failurePolicy: Fail
    workloadSelector:
      matchExpressions:
        - { key: my.vendor/workload, operator: Exists }
    # namespaceSelector: {}      # optional

That is the whole change: a new ValidatingWebhookConfiguration is rendered and its scheduler name is added to the webhook's --external-schedulers union.

Volcano: verified end-to-end

The volcano profile is verified two ways:

1. Against upstream source. schedulerName: volcano is the name Volcano's scheduler answers to and the value its job controller stamps on Pods (createJobPod: pod.Spec.SchedulerName = job.Spec.SchedulerName). volcano.sh/job-name is the label Volcano sets on every workload Pod it creates.
2. With a live kind end-to-end against real upstream Volcano v1.12.2 (installed unmodified from its release YAML) via make e2e-volcano. It:
3. (A) admits a compliant real Volcano Job (its Pod carries volcano.sh/job-name + schedulerName: volcano) through the co-registered validate-volcano.guard.xila.us webhook, and the Pod is actually scheduled by Volcano — reaching Running, bound to a node, with schedulerName=volcano preserved (no scheduler conflict);
4. (B) denies a volcano-labeled privileged Pod and a nodeName-pinned Pod (fail-closed boundary);
5. (C) confirms AND-semantics: Volcano's own admission and our pod webhook are both in effect on the same workload, with no conflict.

Pin a different release with VOLCANO_VERSION=vX.Y.Z make e2e-volcano. Volcano is never modified, rebuilt, or imported.

Operator steps (one or two changes)

1. Provide webhook TLS: either a manual kubernetes.io/tls Secret + enforcement.webhook.tls.caBundle (base64 CA), or set enforcement.webhook.certManager.enabled=true to have cert-manager issue and inject it.
2. Enable one or more profiles and install / upgrade:

helm upgrade --install xila-guard charts/xila-guard -n xila-guard-system --create-namespace \
  --set enforcement.webhook.tls.caBundle="$(base64 -w0 < ca.crt)" \
  -f my-orchestrator-profiles.yaml

If enforcement.enabled=false (the default) the chart renders the minimal webhook these integrations need (ServiceAccount + read-only RBAC + Deployment + Service + TLS) — not the controlled scheduler. If enforcement.enabled=true, the existing enforcement webhook is reused and simply gains the --external-schedulers union.

Default-off guarantee

orchestratorIntegrations defaults to []: no webhook, no arg, nothing renders — helm template is byte-identical to a release without this feature. When enabled, it is fail-closed (failurePolicy: Fail). Local proofs: scripts/orchestrator-smoke.sh (lighter kind proof) and make e2e-volcano (real Volcano e2e). See the orchestrator integration docs.