Kubernetes-native data pipelines for security telemetry at scale

This is a design guide, not a war story. It describes how to build a class of system: a Kubernetes-native pipeline for high-volume security and event telemetry. Everything here is the general pattern and the reasoning behind it, at the level an engineer needs to design their own. It names technologies and tradeoffs, not any organization's deployment.

Security telemetry has three properties that shape the whole design. It is high-volume, often billions of events. It is bursty, with load that can multiply in seconds when something is happening. And it is time-sensitive on one path and not on another: detection wants fresh data now, while investigation and hunting want deep history that can wait. A good design serves both without paying for peak capacity around the clock.

Ingestion

The front door has one job: accept everything, lose nothing, and never become the bottleneck. The pattern is a durable, partitioned log at the boundary, a distributed commit-log system such as Kafka or an equivalent, sitting between the producers and everything downstream.

Three properties make this work at telemetry scale.

• Decoupling. Producers write to the log and move on. If a downstream processor slows down or restarts, the log holds the data. The failure of a consumer is not the loss of an event.
• Partitioning for parallelism. Throughput scales with partitions. Partition on a key that spreads load evenly and keeps related events together, so consumers can be added horizontally as volume grows.
• Replay. A durable log lets you reprocess history. When a detection rule changes or a parser has a bug, you replay the affected window rather than losing the ability to correct the record.

Keep the ingestion tier thin. Validation, light normalization, and schema tagging belong here. Heavy enrichment and analytics belong downstream, where they can scale independently and where a slow enrichment step cannot back-pressure the front door.

Stream and batch, and the backpressure question

The instinct to make everything a stream is a trap, and so is the instinct to batch everything. The right answer is to split the workload by latency requirement and use each model where it fits.

Stream processing handles the detection path, where freshness is the product. A stream processor consumes from the log, applies stateful logic such as windowed aggregations and correlation across events, and emits detections in near real time. The hard part of stream processing is not the happy path. It is backpressure: what happens when input outruns the processor.

Handle backpressure explicitly rather than hoping it does not happen.

• Let the durable log absorb the burst. The processor reads at its own pace and the log holds the backlog, which converts a throughput spike into a latency increase rather than data loss.
• Scale consumers on lag, not on CPU. The signal that matters is how far behind the processor is, measured as consumer-group lag. That is the number that should drive autoscaling.
• Shed or downgrade the low-value path first when saturated, so the detection path keeps its latency budget while bulk archival tolerates delay.

Batch processing handles the investigation path. Large periodic jobs compact, enrich, and reshape history for efficient query. Batch tolerates delay, so it can run on cheaper, interruptible capacity and can be sized for throughput rather than latency.

Autoscaling for spiky load

Security telemetry does not have a steady load you can provision for. It has a baseline and it has events, and the ratio between them is large. Provisioning for peak wastes money most of the time. Provisioning for baseline drops data when it matters most. Autoscaling on the right signals is the way out, and it works at two layers.

Workload autoscaling reacts to demand signals that reflect the actual work. For a telemetry pipeline the meaningful signal is queue depth or consumer lag, not pod CPU. An event-driven autoscaler such as KEDA scales consumers on the lag of the log itself, so the number of processors tracks how much unprocessed data is waiting. When a burst hits, lag rises, consumers scale out, lag drains, consumers scale back in.

Node autoscaling supplies the machines those pods need, quickly. A just-in-time node provisioner such as Karpenter watches for unschedulable pods and brings up right-sized capacity in response, then consolidates and removes nodes when the burst passes. Two levers make this affordable without risking the critical path.

• Run the bulk, interruptible work on spot or preemptible capacity, and keep the detection path on stable capacity. The pipeline's decoupling makes interruptions on the batch path harmless.
• Right-size at provision time. A just-in-time provisioner can pick instance shapes that fit the pending pods rather than padding a fixed node group, which is where a large share of the waste in a static cluster hides.

flowchart LR
  PROD[Telemetry producers] --> LOG[(Durable partitioned log)]
  LOG --> STREAM[Stream processor
detection path]
  LOG --> BATCH[Batch jobs
enrich and compact]
  STREAM --> HOT[Recent-data store]
  BATCH --> LAKE[(Object store
columnar tables)]
  HOT --> Q[Query engine]
  LAKE --> Q
  Q --> USERS[Detection, hunting, investigation]
  LAG[Consumer lag] -.scales.-> STREAM
  PENDING[Unschedulable pods] -.provisions nodes.-> NODES[Just-in-time capacity]
            

The query and analytics layer

Storing telemetry is easy. Making years of it queryable at a cost you can defend is the real design problem. The pattern that scales is separation of storage from compute, built on an object store.

Land the deep history as columnar files in object storage, organized as open table formats such as Apache Iceberg or an equivalent, which add table semantics, schema evolution, and time travel over plain files. Point a distributed SQL engine such as Trino or an equivalent at those tables for interactive query. The properties that matter.

• Storage and compute scale independently. History grows in cheap object storage without adding query capacity. Query capacity spins up for a hunt and spins down after, without moving data.
• Columnar plus partition pruning. Security queries filter hard on time and a few dimensions. Columnar layout and partitioning let the engine read only the relevant slice, which is the difference between a query that scans a day and one that scans a decade.
• Open formats keep the data portable. The tables are not locked to one engine. That is both an operational safeguard and, for a government customer, a guarantee that the data outlives any single tool. It is the built-to-leave principle applied to the data layer.

Keep a smaller, faster store for the recent window that detection and triage hit constantly, and let the object-store-backed tables serve the long tail. Most reads are recent. Most data is old. Serve each from the tier that fits.

Networking and DNS at cluster scale

At high fan-out, the parts of Kubernetes that are invisible on a small cluster become the parts that page you. Two deserve explicit design.

DNS. A pipeline with many short-lived pods resolving service names generates a surprising volume of DNS queries, and cluster DNS can become a contention point that shows up as mysterious tail latency everywhere. Design for it: run a node-local DNS cache so most lookups never leave the node, tune negative-cache and TTL behavior for the workload, and watch DNS as a first-class signal rather than discovering it during an incident.

East-west throughput and policy. The internal traffic between ingestion, processing, and storage is the bulk of the network load, and it has to be both fast and governed. Choose a CNI that carries the throughput without becoming the bottleneck, and express segmentation as network policy so the pipeline's tiers can only talk to the neighbors they should. Fast and open is a liability. Fast and segmented is the goal.

Cost control as a design property

A telemetry platform's cost is a design outcome, not a billing surprise. The levers are built in, not bolted on.

• Tier the storage. Recent data on fast storage, deep history on cheap object storage, lifecycle rules moving data down the tiers automatically. The cost curve should follow the access curve.
• Scale to the work, not to the peak. Lag-driven and pending-pod-driven autoscaling means you pay for capacity while there is work and release it when there is not. The static cluster sized for the worst hour is the most common source of waste.
• Interruptible capacity for delay-tolerant work. The batch and archival paths run on spot or preemptible nodes because the pipeline's decoupling makes an interruption a non-event.
• Attribute the cost. Instrument spend by pipeline stage so the expensive component is visible and can be tuned, rather than hidden inside a single large bill.

Observability and SLO design

A data platform's users do not experience uptime. They experience freshness and completeness. Design the SLOs around what they actually depend on.

• Freshness. End-to-end latency from event produced to event queryable on the detection path. Measured continuously, alerted when it exceeds the budget. This is usually the SLO that matters most.
• Completeness. The fraction of produced events that made it through, measured by reconciling counts across the pipeline. Silent loss is the worst failure mode for a security pipeline, so instrument for it deliberately.
• Query availability and latency. Whether the interactive engine answers, and how fast, for the query shapes that matter.

Instrument every stage: ingestion rate, consumer lag, processing latency, batch job duration and success, storage growth, query performance. Consumer lag deserves special attention, because it is the earliest honest signal that the pipeline is falling behind, well before users notice stale data. When these signals are wired into the SLOs, the platform tells you it is degrading before anyone has to report that it did.

Designing the class, not the instance

The shape holds across environments: a durable log at the front, a split between a fresh detection path and a deep investigation path, autoscaling driven by the honest signals of lag and pending work, an object-store-backed analytics layer in open formats, and SLOs written around freshness and completeness. What changes between deployments is the sizing and the specific components, not the shape. That is why this is a reference architecture. It is the design you start from, adapted to the constraints in front of you, built to run on infrastructure the customer owns and to be handed over when the work is done.