Secure agentic AI on Kubernetes in a restricted environment

An autonomous agent is a strange thing to secure. It is a workload you deploy on purpose, and it is an adversary you have to plan for, at the same time. It reads untrusted input, decides what to do, and reaches for tools that touch real systems. In a connected environment you can lean on external services to watch it. In an air-gapped or impact-level-restricted environment you cannot. Everything that governs the agent has to live inside the boundary with it, on infrastructure the customer already owns.

This is a reference design for that setting. It assumes a Department of War or federal mission where the cluster is government-owned, egress is denied by default, and every control has to produce evidence an authorizing official will read. The goal is not a clever agent. The goal is an agent whose blast radius is bounded by design, and a system the customer can operate after we leave.

The constraints set the design

Three constraints drive every decision below.

• No egress. The cluster cannot call a hosted model API, a public package registry, or an external policy service. Inference, tools, and controls all run in-cluster or at a controlled gateway.
• Least privilege is not optional. The agent gets the narrowest set of credentials and network paths that let it do its job, scoped per task, expiring on a short clock. No standing admin, no shared service account.
• Evidence is a deliverable. Every control maps to a named requirement in an RMF authorization narrative. If a control cannot produce a log, an attestation, or a signed artifact, it does not count.

The rest of this design is what those three constraints force you to build.

Cluster topology

Segment the cluster by trust, not by team. Four namespaces, with network policy denying traffic between them unless a rule allows it.

• Inference. The model server and its GPUs. It accepts requests only from the agent gateway. It has no egress and no cluster-API access.
• Agent runtime. The orchestration layer and the running agents. It can reach the gateway and nothing else on the network.
• Gateway. The single policy enforcement point. Every tool call and every model call passes through it. This is the narrow waist of the system.
• Platform. Identity, secrets, admission control, logging, and the policy engine. The agent runtime cannot reach into it directly.

The value of this layout is that the interesting traffic has exactly one path. An agent that wants to call a tool or the model has to go through the gateway, because network policy gives it no other route. You are not hoping the agent behaves. You are removing the option.

flowchart TB
  subgraph AR[Agent runtime namespace]
    ORCH[Orchestration and running agents]
  end
  subgraph GW[Gateway namespace]
    PEP[Agent gateway and policy enforcement point]
  end
  subgraph INF[Inference namespace]
    MODEL[In-cluster model server and GPUs]
  end
  subgraph PLAT[Platform namespace]
    IDP[Workload identity]
    POL[Policy engine]
    LOG[Audit and behavior logs]
    ADM[Admission control]
  end
  TOOLS[Approved tool endpoints]
  ORCH -->|all model calls| PEP
  ORCH -->|all tool calls| PEP
  PEP -->|scoped, logged| MODEL
  PEP -->|scoped, logged| TOOLS
  PEP -.consults.-> POL
  PEP -.short-lived creds.-> IDP
  PEP -->|decisions and traces| LOG
  ADM -.gates images and skills.-> AR
            

Where the model lives

In a restricted environment the model runs in-cluster. That decision is made for you by the no-egress constraint, but it has design consequences worth stating.

The inference namespace holds the model server and its accelerators. It is deliberately dumb. It does not know which agent is calling, it does not hold credentials for downstream systems, and it cannot reach the network beyond the gateway. All of the judgment about whether a given call is allowed happens before the request reaches the model, at the gateway. Keeping the model server free of policy logic means you can swap models, quantize them, or run more than one behind the gateway without touching the control plane.

Write the design so the model is a component, not the center. The agent runtime addresses the model through the gateway using a stable internal interface. If the mission constraints require a specific model, or forbid a specific vendor, that is a configuration at the gateway, not a rewrite. This is the model-agnostic posture: the controls do not depend on which model answers.

The agent gateway

The gateway is the most important component in the system, because it is the one place where a request stops being a suggestion and becomes an authorized action. Two request types flow through it: model calls and tool calls. It handles them differently.

Model calls get input and output mediation. On the way in, the gateway can strip or tag untrusted content so the runtime knows which parts of a prompt came from a tool result versus an operator. On the way out, it inspects tool-call requests the model emits before they are ever dispatched.

Tool calls get the real enforcement. Before a tool runs, the gateway asks the policy engine a specific question: is this agent, on this task, with this identity, allowed to call this tool with these arguments right now. The policy is data, versioned in git, not code buried in the runtime. A representative rule reads like this.

action: tool.call
match:
  tool: filesystem.write
  path: "/workspace/**"        # scoped to the task workspace
require:
  identity: task-bound          # short-lived, one task, one agent
  scope: [workspace-write]
effect: allow
# any path outside the task workspace falls through to deny

Deny is the default. A tool the policy does not name cannot be called. Arguments that fall outside the allowed shape are rejected before dispatch, not after. And because the gateway is the chokepoint, the decision is logged with the full context: which agent, which task identity, which tool, which arguments, allowed or denied, and why.

Identity and secrets

Standing credentials are the failure mode that turns a contained agent into a breach. The design removes them.

Each agent task runs under a workload identity issued by the platform, scoped to that task, valid for minutes rather than days. When the gateway dispatches a tool call, it exchanges the task identity for a short-lived credential for that specific downstream system, with that specific scope. The credential expires on its own. There is no long-lived secret sitting in an environment variable for an attacker to read, and no shared service account whose compromise unlocks everything.

The practical version of this is a workload-identity provider issuing signed tokens with a tight audience and a short expiry, a secrets manager that mints per-task credentials on request, and a hard rule that agents never see a raw secret. They see a capability, brokered by the gateway, that stops working shortly after the task ends.

Network policy and egress control

Air-gapped does not mean unplugged. It means every allowed path is named and everything else is denied. Kubernetes network policy expresses this well when you write it as an allowlist.

• The agent runtime may open connections to the gateway. Nothing else.
• The gateway may reach the inference service and the specific, approved tool endpoints. Nothing else.
• The inference namespace may accept from the gateway and originate nothing.
• A default-deny policy sits under all of it, so a new workload with no matching rule has no network at all until someone writes one.

The point of default-deny is that a mistake fails closed. If an agent is coaxed into reaching for an endpoint that is not on the list, the connection never opens, and the CNI that enforces the policy logs the denied attempt: NetworkPolicy declares the rules, and a CNI with flow logging supplies the evidence. Containment does not depend on the agent being well-behaved.

Supply chain for skills and images

An agent's capabilities arrive as two kinds of artifact: the container images that run the platform, and the skills or tools the agent can invoke. Both are supply-chain surface, and both are gated the same way.

Every image is built in a secure software factory pattern, signed at build time, and accompanied by a software bill of materials. Skills carry provenance and a signature from an approved source. At deploy time, an admission controller refuses anything that is unsigned, fails policy, or carries a bill of materials with a disallowed component. This is where DoW-accredited container hardening pipelines earn their place: the hardening and the evidence are produced by the pipeline, not asserted after the fact.

policy: admission.image
require:
  signature: verified          # signed by an approved key
  sbom: present                # bill of materials attached
  attestation: build-provenance
deny:
  components: [known-prohibited]
effect: allow-if-all-met

The reason to enforce this at admission, rather than in a scanner that runs later, is that later is too late. An unsigned skill that reaches the runtime is a skill you now have to reason about. Blocking it at the door means you never have to.

Observability for agent behavior

Traditional workload monitoring watches CPU, memory, and error rates. Agent monitoring has to watch decisions. The signal you care about is behavioral: which tools an agent reached for, in what order, with what arguments, and whether the pattern matches what this class of task is supposed to do.

Because every model call and tool call passes through the gateway, the gateway is also your instrumentation point. It emits a structured trace per task: the sequence of tool calls, the policy decisions, the identities used, and the results. That trace is the raw material for two things. Live, it feeds anomaly detection that can quarantine a task whose behavior drifts from its expected shape. After the fact, it is the audit record that shows an assessor exactly what the agent did.

Prefer containment over hard-kill as the default response. A task that trips a behavioral rule gets its capabilities narrowed or paused for review, rather than terminated outright, so an operator can see what happened and the system degrades gracefully instead of thrashing.

Authorization evidence, produced not assembled

The controls above are not just security. Each one is a sentence in an authorization narrative, and the system emits the proof as it runs.

The difference this makes is timing. When the authorization package is due, the evidence already exists, because the running system has been producing it the whole time. This is the same idea Helocast applies across a platform, covered in depth in the reference architecture on producing ATO evidence as a byproduct of platform engineering.

What the customer owns when we leave

The measure of this design is what remains after the engagement ends. Helocast builds it, documents it, trains the customer's engineers to run it, and extracts cleanly.

What stays is a self-contained system: the cluster, the gateway and its policy set, the identity and secrets brokering, the network policies, the admission controls, the behavioral tracing, and the authorization evidence, all running on infrastructure the government already owns, built from open components the government keeps. There is no Helocast-hosted service in the path, no license that expires, and no dependency on us to operate it. The policies are readable and editable by the customer's team. The evidence keeps generating itself. That is the built-to-leave model applied to the hardest workload in the building.