Architecting a Kubernetes AI platform for CNCF conformance

Every vendor selling an "AI-ready" Kubernetes means something different by it. One ships a GPU operator and calls it done. Another bakes in a specific inference server and a specific scheduler and assumes you want both. A workload that trains on one platform does not always run on the next, because the accelerator interface, the scheduling contract, and the inference routing are all subtly different. That fragmentation is the tax you pay for building AI on Kubernetes without a shared definition of what an AI platform must do. The CNCF Certified Kubernetes AI Conformance program exists to remove that tax: it defines a baseline set of capabilities an AI platform must expose, so that a training job or an inference service written against the standard runs the same way on any conformant platform.

This piece describes how a curated, open-source stack is architected to meet the v1.35 requirement set, mapped requirement by requirement to the component that satisfies it, and how that platform would move through certification. It is a reference architecture and a certification methodology, not a claim of a certificate already in hand. The platform is architected to meet the requirements and follows the certification path described below; it is not asserted as CNCF-certified in the past tense, because certification is a factual status and this piece does not invent one.

One distinction has to be clear before anything else, because a knowledgeable reader will catch a conflation of it instantly. CNCF AI Conformance certifies AI-workload portability and interoperability: that AI and ML workloads run according to the standard. It does not certify security posture, hardening, or authorization-readiness. The platform's security hardening and authorization-evidence capability is a separate layer Helocast adds, and it is not something the CNCF program certifies. Throughout this piece those are two distinct claims: conformant for AI-workload portability, and separately, hardened and authorization-evidence-ready by us. They are never the same claim.

What CNCF AI Conformance actually is

The Certified Kubernetes AI Conformance program launched in November 2025. It certifies that a platform exposes a defined set of AI-platform capabilities: Dynamic Resource Allocation for accelerators, gang scheduling for distributed jobs, Gateway API routing for inference, per-accelerator and per-service metrics, isolated accelerator access, and the ability to run a complex AI operator with its own custom resources. A workload author who targets those capabilities can move between conformant platforms without rewriting the parts that touch the platform.

It builds on top of base Kubernetes conformance. A platform must already be Certified Kubernetes conformant, the long-running program that verifies a distribution implements the core Kubernetes APIs correctly, before it can be certified for AI. The AI submission references the base conformance record directly.

Certification today is a structured self-assessment reviewed by CNCF, not an automated test result. You meet the requirements, complete the conformance checklist, gather public evidence for each item, and open a pull request against the cncf/k8s-ai-conformance repository. CNCF reviews the submission, typically within about ten business days. Automated conformance tests are planned for 2026 but are not the mechanism today, so an honest account says "submitted a reviewed self-assessment," not "passed an automated test suite" that does not yet exist.

A certification is valid for one year and must be renewed. It is granted per product and per configuration. This last point is the one that matters most for the government work below: a cloud deployment and an air-gapped deployment of the same platform are certified as two distinct configurations, each with its own submission and its own evidence. The requirement set tracks the Kubernetes release cadence, with tracks spanning v1.33 through v1.36; this piece maps against the v1.35 set, which is twelve requirements: nine at level MUST and three at level SHOULD.

The reference architecture

The stack is a set of open components chosen so that each conformance requirement has a clear owner, layered from the base distribution up through the operators, with a security overlay drawn distinctly on top. The overlay is the part CNCF does not certify, and the diagram keeps it visually separate for exactly that reason.

flowchart TB
  subgraph SEC[Security and evidence overlay, not CNCF certified]
    SIGN[Signed images and SBOMs]
    ADM[Admission control]
    NP[Network policy, default-deny]
    EV[Authorization evidence, 800-53 mapping]
  end
  subgraph OP[Operator and serving layer]
    RAY[KubeRay]
    TRN[Kubeflow Trainer]
    SRV[KServe and vLLM serving]
  end
  subgraph NET[Networking and inference layer]
    GW[Gateway API]
    GIE[Gateway API Inference Extension]
  end
  subgraph SCH[Scheduling layer]
    GANG[Kueue and Volcano, gang scheduling]
    AS[Cluster Autoscaler or Karpenter, HPA]
  end
  subgraph OBS[Observability layer]
    DCGM[DCGM Exporter]
    PROM[Prometheus and Grafana]
  end
  subgraph ACC[Accelerator layer]
    GPUOP[NVIDIA GPU Operator]
    DRA[DRA driver, resource.k8s.io]
    SHARE[MIG and time-slicing]
  end
  subgraph BASE[Base layer]
    K8S[Certified Kubernetes distribution]
  end
  SEC -.hardens and observes.-> OP
  OP --> NET
  OP --> SCH
  NET --> ACC
  SCH --> ACC
  OBS -.scrapes.-> ACC
  OBS -.scrapes.-> OP
  ACC --> BASE
            

The base layer is a Kubernetes distribution able to pass base conformance. The accelerator layer is the NVIDIA GPU Operator managing the driver, container toolkit, device plugin, metrics exporter, and MIG configuration through a single declarative custom resource, with the Dynamic Resource Allocation driver exposing accelerators through the newer DRA APIs. The scheduling layer pairs a gang scheduler with node and pod autoscaling. The networking layer is the Gateway API plus its inference extension. The observability layer scrapes both the accelerators and the serving workloads into Prometheus. The operator layer runs a real distributed-AI operator with its own custom resources. Each choice is justified against its requirement in the mapping below.

Requirement by requirement

This is the technical spine. Each of the twelve v1.35 requirements maps to a named component, a reason that component satisfies it, and the evidence that would go in the checklist. The table states the mapping; the prose after it explains the MUST requirements, which are the ones that gate certification.

Accelerators

dra_support (MUST) is satisfied by the platform serving the Dynamic Resource Allocation APIs under resource.k8s.io, which reached general availability in Kubernetes v1.34, together with the NVIDIA DRA driver. DRA is the move away from requesting accelerators as opaque integer counts and toward structured, claim-based requests: a workload asks for a device with the properties it needs, and the driver publishes what the node actually has. The evidence is that the API responds and that a sample claim schedules onto a real device.

apiVersion: resource.k8s.io/v1
kind: DeviceClass
metadata:
  name: gpu.example
spec:
  selectors:
    - cel:
        expression: device.driver == "gpu.nvidia.com"
---
apiVersion: resource.k8s.io/v1
kind: ResourceClaimTemplate
metadata:
  name: single-gpu
spec:
  spec:
    devices:
      requests:
        - name: gpu
          exactly:
            deviceClassName: gpu.example

A DeviceClass plus a ResourceClaimTemplate. The workload references the template; the scheduler and the DRA driver do the matching. This is the fine-grained request model the requirement asks for.

The three SHOULD requirements sit on the same accelerator layer. driver_runtime_management is met by the NVIDIA GPU Operator, which installs and continuously reconciles the driver, container toolkit, device plugin, and metrics exporter from a single ClusterPolicy, giving a verifiable mechanism that the right versions are present on accelerator nodes. gpu_sharing is met with two strategies: MIG hardware partitioning, which carves a physical accelerator into isolated slices exposed as schedulable resources, and software time-slicing, which oversubscribes a device across pods without memory or fault isolation. Offering at least one satisfies the requirement; the platform offers both and documents the isolation tradeoff. virtualized_accelerator covers vGPU-style virtualization. On a bare-metal government cluster that presents physical accelerators directly, virtualization is often not in play, and the correct checklist answer is a justified N/A: not "we do not support it," but a written explanation that the configuration presents physical accelerators with MIG for partitioning, so the vGPU context does not apply. The program permits N/A only with that kind of justification, and this is where it is used honestly.

Networking and inference

ai_inference (MUST) requires a Kubernetes Gateway API implementation with advanced traffic management for inference. The base Gateway API provides weighted traffic splitting and header-based routing, which covers canarying a new model version and routing on protocol headers. The Gateway API Inference Extension adds the inference-aware piece: an InferencePool that groups model-serving endpoints and an Endpoint Picker that routes a request to the right replica based on the model named in the request body, alongside an InferenceObjective that expresses serving priorities. Implementations that support the extension include Envoy AI Gateway, Istio, kgateway, and NGINX Gateway Fabric; the platform picks one and serves real inference behind it, with KServe or vLLM as the model server. The evidence is a working weighted split and a working model-aware route, with the route manifests attached.

Scheduling and orchestration

gang_scheduling (MUST) is the all-or-nothing property that distributed training needs: a job that requires eight workers should get eight or none, never four that sit idle holding accelerators while they wait for the rest. The platform runs Kueue for gang admission and quota and Volcano, a CNCF project, for gang placement; the two are commonly paired, with Kueue deciding when a job is admitted and Volcano ensuring the whole pod group lands together. The requirement asks the platform to demonstrate at least one such solution operating; the evidence is a multi-pod job admitted atomically, shown in the scheduler logs.

cluster_autoscaling (MUST) and pod_autoscaling (MUST) are both written conditionally in the requirement text. The level is a plain MUST, but each opens with an "if": if the platform provides a cluster autoscaler or an equivalent mechanism, it must scale accelerator node groups up and down based on pods pending on those accelerators; if the platform supports the HorizontalPodAutoscaler, that must work for accelerator-using pods, including scaling on custom AI and ML metrics. Node scaling is met with Cluster Autoscaler or Karpenter against the accelerator node group. Pod scaling is met with the HorizontalPodAutoscaler driven by a custom metric surfaced through the Prometheus Adapter or KEDA, so an inference deployment can scale on queue depth or tokens per second rather than CPU. The conditional framing matters in the air-gapped configuration below, where a fixed-capacity enclave may have no elastic node pool at all.

Observability

accelerator_metrics (MUST) is satisfied by the NVIDIA DCGM Exporter, which publishes per-accelerator utilization and memory in Prometheus exposition format at a scrapable endpoint, as a DaemonSet, with pod attribution through the kubelet pod-resources API. Where the hardware exposes them, it also publishes temperature, power draw, and interconnect bandwidth, which the requirement asks for when available. Alignment with OpenTelemetry is handled by scraping the endpoint with an OpenTelemetry Collector where a single telemetry pipeline is wanted. ai_service_metrics (MUST) is the workload side of the same idea: a monitoring system that discovers and collects metrics from AI workloads exposing them in a standard format. Prometheus scrapes the serving metrics that vLLM, KServe, and Triton already emit in exposition format, and Grafana renders them. The evidence for both is a metrics endpoint discovered and collected, with the scrape configuration attached.

Secure accelerator access

secure_accelerator_access (MUST) requires that access to accelerators from inside containers is isolated and mediated by the Kubernetes resource-management framework, the device plugin or DRA, and the container runtime, so one workload cannot reach another's device or interfere with it. This is a platform capability requirement, and it is worth being precise about what it is not: it is one interoperability requirement about device isolation, not a security certification of the platform. The isolation is demonstrated by showing that a pod granted a device cannot see a device it was not granted, and that the runtime and the resource framework, not ad hoc container configuration, are what enforce it. The broader hardening posture is the separate layer described later, and conflating the two is exactly the error this piece avoids.

Operator

robust_controller (MUST) asks the platform to prove it can run a complex AI operator with a custom resource definition, reliably: pods run, admission webhooks are operational, and custom resources reconcile. The platform runs KubeRay, the Ray operator, or Kubeflow Trainer, the successor to the older training operator with its TrainJob API, either of which installs a controller, serves validating and mutating webhooks, and reconciles its custom resources into running distributed jobs. This requirement is the integration test for the whole stack: a training job submitted through the operator exercises the accelerator layer, the gang scheduler, and the metrics pipeline at once. The evidence is the operator installed, its webhooks serving, and a custom resource reconciled to completion.

Base Kubernetes conformance comes first

AI conformance is not a standalone certification. The platform must already be Certified Kubernetes conformant, which is the older and larger program that verifies a distribution implements the core Kubernetes APIs the way upstream defines them. That base certification is obtained by running the upstream conformance test suite against a live cluster, commonly executed with Sonobuoy, and submitting the passing results to the cncf/k8s-conformance repository, which lists the platform as conformant. The AI conformance submission then references that base record by URL. Getting base conformance in place first is not a formality; it is the ground the AI requirements stand on, and the AI review will look for the base listing.

The certification path

The path from a working platform to a listed AI conformance record has four steps, and none of them involves a test result the program does not yet produce.

flowchart TB
  BASE[Base Kubernetes conformance listed] --> PREP[Prepare, review the requirements]
  PREP --> DOC[Document, complete the checklist and gather public evidence]
  DOC --> SUB[Submit a PR to cncf k8s-ai-conformance]
  SUB --> REV[CNCF review, about ten business days]
  REV --> CERT[Listed as conformant, valid one year]
  CERT -.renew annually.-> PREP
  DOC -.unreleased product.-> PRIV[Private review by email to CNCF]
            

Prepare is reviewing the requirements and the program terms. Document is the work: complete the AIConformance-1.35.yaml checklist, which lists every requirement with its id and level, and record for each a status, a piece of publicly reachable evidence, and notes. The submission itself is a per-product manifest placed under the version directory in the repository, carrying product metadata and the completed requirements. The evidence for every MUST has to resolve to something a reviewer can open: a documentation page, a manifest, a captured test output.

# Per-product conformance submission (illustrative)
vendorName: Example Integrator
platformName: Example AI Platform
platformVersion: "1.0"
kubernetesVersion: "1.35"
websiteUrl: https://example.gov
documentationUrl: https://docs.example.gov/ai
productLogoUrl: https://example.gov/logo.svg
k8sConformanceUrl: https://github.com/cncf/k8s-conformance/pull/0000
contactEmailAddress: platform@example.gov
requirements:
  - id: dra_support
    level: MUST
    status: met
    evidence: https://docs.example.gov/dra
    notes: DRA served at resource.k8s.io; sample claim scheduled
  - id: gang_scheduling
    level: MUST
    status: met
    evidence: https://docs.example.gov/gang
    notes: Kueue admits atomically; Volcano places the pod group
  - id: virtualized_accelerator
    level: SHOULD
    status: not-applicable
    evidence: https://docs.example.gov/accelerators
    notes: Bare-metal config presents physical accelerators; MIG used for partitioning

The submission manifest. Every MUST carries a status, public evidence, and notes; a SHOULD may be N/A with a real justification, as shown for virtualized accelerators on bare metal.

Submit is opening a pull request against cncf/k8s-ai-conformance, with the product logo supplied in a vector format. Review is CNCF checking the submission, typically within about ten business days. For a product that is not yet public, the program supports a private review: rather than opening the pull request, the submitter emails a packaged copy to conformance@cncf.io. The listing is valid for one year and is renewed by resubmitting against the current requirement track. It is worth stating plainly that this is a reviewed self-assessment. The platform meets the requirements and the submission documents how, with public evidence, and CNCF reviews it. It is not an automated pass, because the automated tests are planned and not yet the mechanism.

The air-gapped configuration

Because conformance is granted per configuration, an air-gapped deployment is its own certification with its own submission, and this is where the architecture earns its keep. Almost nothing in the requirement set assumes internet access; what changes is how every image, chart, driver, and model artifact gets into the enclave, and how a few requirements are answered when there is no cloud API to call.

The mechanics are registry and mirror work. A central registry inside the boundary, commonly Harbor, holds the mirrored images and charts; per-site pull-through registries such as Zot serve the workload clusters, which reach only their local mirror. Images move in with skopeo, oras, or crane; Helm charts move as OCI artifacts with their image references rewritten to the internal registry. In a government enclave the transfer itself is a controlled procedure: media is wiped, written, hashed, sealed, re-hashed on the far side, staged into the registry, scanned, and only then promoted. None of that is exotic; it is the standard shape of a declarative, GitOps-deployed, hardened Kubernetes platform operating without egress.

# GPU Operator, air-gapped values
driver:
  repository: registry.enclave.local/nvidia
  repoConfig:
    configMapName: local-package-mirror   # offline driver packages
  certConfig:
    name: enclave-ca                       # self-signed trust
toolkit:
  repository: registry.enclave.local/nvidia
devicePlugin:
  repository: registry.enclave.local/nvidia
dcgmExporter:
  repository: registry.enclave.local/nvidia
# every image resolves inside the boundary, no external registry egress

The GPU Operator pointed entirely at the in-enclave registry, with driver packages served from a local mirror and a self-signed trust anchor. The same pattern, an internal repository plus mirrored images, applies to every component in the stack.

Two requirements need a specific answer offline. Model weights never come from a public model hub at runtime; they are staged into the enclave ahead of time and served from a persistent volume populated by a one-time copy job, or from an in-enclave object store, so the serving path that satisfies ai_inference and ai_service_metrics has its artifacts local. And cluster_autoscaling is the requirement most likely to change: a fixed-capacity enclave with a static set of bare-metal accelerator nodes may have no elastic node pool to scale, in which case the honest checklist answer is a justified N/A explaining that the configuration has fixed node capacity, while pod_autoscaling still applies within that capacity. Base conformance is equally achievable offline, because the upstream test suite runs against the local cluster once its images are staged.

The certification claim for this configuration is the same shape as any other: conformant for AI-workload portability. What the mission additionally requires, and what CNCF conformance does not speak to, is the hardening and the authorization evidence, which is the next section and a separate claim.

The security and authorization-evidence layer, a separate claim

Everything above earns a conformance listing. None of it hardens the platform or proves anything to an authorizing official, and it is important not to let the conformance work imply that it does. The security and authorization-evidence layer is a distinct capability Helocast adds on top, and it is not what the CNCF program certifies.

The hardening approach is the familiar supply-chain and admission posture, described generically because the pattern, not any specific product, is what matters: images built in a secure factory, signed at build time, and accompanied by a software bill of materials; an admission controller that refuses anything unsigned, unattested, or carrying a disallowed component; network policy that denies by default so a workload with no matching rule has no network; and provenance that traces every artifact to a build it can name. This is the declarative, GitOps-deployed, hardened platform model, and the hardening is produced by the pipeline as evidence rather than asserted afterward.

The authorization-evidence side maps each control to the NIST 800-53 families it satisfies and emits the proof continuously as the platform runs, so the monitoring signals an assessor wants are a byproduct of operation rather than a separate document assembled at the end. The result is a continuous-monitoring posture that supports a path to continuous authorization, covered in depth in the reference architecture on producing ATO evidence as a byproduct of platform engineering. Stated as plainly as possible: CNCF conformance attests that the AI workloads run portably; this layer attests that the platform is hardened and produces authorization evidence. They are two different attestations from two different authorities, and a government reader is right to expect them kept apart.

What the customer owns when we leave

The point of building on open components and a documented certification path is that the result belongs to the customer, not to us. What remains after the engagement is a platform that is conformant for AI-workload portability and hardened for the mission, running on infrastructure the government already owns, built from components the government keeps.

The conformance submission and its evidence are handed over as artifacts the customer's team maintains: the completed checklist, the per-product manifest, the public documentation each requirement points to, and the base conformance record underneath. Renewal is theirs to run, once a year, against the current requirement track. The authorization artifacts are handed over the same way. There is no Helocast-hosted service in the path, no license that expires, and no dependency on us to operate or recertify the platform. The engineers we train can edit the manifests, regenerate the evidence, and resubmit. That is the built-to-leave model applied to a platform that has to satisfy two independent standards at once.

What this demonstrates

A conformance listing is a narrow, factual claim: these AI workloads run portably on this platform. The reason it is worth pursuing is what it proves about the team behind it. Architecting a stack that meets every one of the twelve requirements, taking it through base conformance and then AI conformance, and doing it in an air-gapped government configuration with a hardening and authorization-evidence layer on top, is a demonstration that the team can build and deliver a real Kubernetes AI platform in the hardest environment there is.

The same discipline shows up across the rest of this work: the secure agentic AI reference architecture for running autonomous agents on a restricted cluster, the telemetry pipeline design for the observability that feeds it, and the authorization-evidence architecture that turns the whole platform into its own proof. Conformance is not the product. It is evidence that the platform underneath it was built correctly, by people who can hand it back and walk away.