Why Most Teams Have an Uneven Security Stack
The pattern we see in most mid-size Kubernetes environments follows a predictable gradient. The build layer is well-covered — Trivy, Grype, or Snyk runs in CI, CVE scores gate image promotion, and SAST tools scan application code before it reaches a container. The deploy layer is partially covered — admission controllers exist, maybe OPA Gatekeeper or Kyverno is in place, PodSecurityStandards are enforced in some namespaces. The runtime layer is a gap — no eBPF agent, no syscall visibility, Falco maybe running but untouched since initial install, logging shipped to CloudWatch or Elasticsearch but nobody running regular queries against it for threat patterns.
This isn't negligence. It's the natural maturity curve of a K8s security program. Build-time tools integrate cleanly into existing CI/CD workflows and produce actionable developer-facing output (fix this CVE in this dependency). Deploy-time controls fit into the GitOps workflow. Runtime detection requires a different architecture — a persistent agent on every node, an event processing pipeline, a rule evaluation engine, an alert routing layer — and it's harder to justify the investment until after an incident makes the gap visible.
This guide maps all three layers with enough specificity to identify where your gaps actually are, not just where the categories sound unfamiliar.
Layer 1: Build-Time Security
Build-time security covers everything that happens before a container image is pushed to a registry. The tools and checks in this layer operate on static artifacts — source code, dependency manifests, Dockerfiles, Helm chart values files.
Container Image Scanning
Image scanners (Trivy, Grype, Snyk Container) compare the packages installed in an image against CVE databases (NVD, GitHub Advisory, OS vendor advisories). A scanner run against a node:18-alpine base image will identify known CVEs in Alpine packages and Node.js runtime. The coverage is: known vulnerabilities in packages present in the image at scan time.
What image scanning cannot catch: vulnerabilities that don't have CVEs yet (zero-days), malicious code that doesn't exploit a known CVE (supply chain compromise via typosquatting or dependency confusion), runtime misconfigurations that only manifest when the container executes, and behavioral malware that's dormant at scan time. These are not hypothetical edge cases — supply chain attacks against npm, PyPI, and container registries documented throughout 2023-2025 repeatedly demonstrated that clean-CVE images can contain malicious code.
The correct calibration: image scanning is essential and should block image promotion on CRITICAL CVE severity findings. It is not a sufficient condition for "this image is safe to run in production."
Dockerfile and IaC Linting
Checkov, Hadolint, and Semgrep rules for Kubernetes YAML catch configuration problems before they reach the cluster: USER root in a Dockerfile, privileged: true in a pod spec, allowPrivilegeEscalation: true without justification, overly broad RBAC bindings (verbs: ["*"] on resources: ["*"]). These are static checks — they find intent misconfigurations.
The limitation: static analysis can't distinguish between a policy file that declares allowPrivilegeEscalation: false and actual runtime behavior when that container encounters a SUID binary path. Policy intent and runtime reality can diverge.
Dependency SAST
SAST tools (Semgrep, CodeQL, Bandit for Python) scan source code for patterns associated with vulnerabilities: SQL injection, command injection, hardcoded credentials, use of deprecated crypto APIs. For K8s-specific risks, tools like kube-score or pluto identify deprecated API versions in Helm charts that will break on cluster upgrades.
Layer 2: Deploy-Time Security
Deploy-time controls run at the point where a workload is admitted into the cluster. The Kubernetes admission control architecture provides two extension points: ValidatingAdmissionWebhooks (can reject requests) and MutatingAdmissionWebhooks (can modify requests). Built-in PodSecurityAdmission handles the most common security policy enforcement.
PodSecurityStandards
Kubernetes PodSecurityAdmission (replacing the deprecated PodSecurityPolicy in K8s 1.25) implements three profiles at the namespace level via the pod-security.kubernetes.io/enforce namespace label:
- privileged: no restrictions. Never use in production namespaces.
- baseline: blocks the most obviously dangerous configurations: privileged containers, hostPID/hostNetwork/hostIPC, hostPath volumes that escape the container filesystem, most dangerous capabilities. This is a minimum bar for any production namespace.
- restricted: adds require-non-root, disallow privilege escalation, restrict volume types to safe set, drop all capabilities. This is the target profile for stateless application workloads.
Achieving restricted across all production namespaces is the correct goal. It eliminates a large portion of the container escape attack surface and forces privilege escalation attempts to more sophisticated kernel-level vectors. The challenge: operators, system components (cert-manager, CSI drivers, monitoring agents), and anything that mounts hostPath volumes legitimately requires exceptions. Building those exceptions correctly — minimum-capability allowances, specific namespace exemptions rather than cluster-wide relaxation — is where most teams get it wrong.
Policy-as-Code (OPA Gatekeeper / Kyverno)
Beyond PodSecurityStandards, policy engines like OPA Gatekeeper (uses Rego) and Kyverno (uses YAML-based policy language) enforce custom organizational policies: required resource limits on all containers, required image registry allowlists, required labels for team attribution, required network policy presence on every namespace. These fill gaps that PodSecurityAdmission doesn't cover.
A practical Kyverno policy enforcing registry allowlisting:
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
name: restrict-image-registries
spec:
validationFailureAction: enforce
rules:
- name: check-registry
match:
any:
- resources:
kinds: [Pod]
validate:
message: "Image must be from an approved registry."
pattern:
spec:
containers:
- image: "123456789012.dkr.ecr.us-east-1.amazonaws.com/* | ghcr.io/your-org/*"This policy blocks any pod that tries to pull an image not from your ECR account or your GitHub Packages registry — closing a supply chain attack vector where a compromised dependency pulls in an attacker-controlled public image.
NetworkPolicy
Kubernetes NetworkPolicy objects, enforced by the CNI (Calico, Cilium, Antrea), define allowed ingress and egress traffic at the pod level. Default deny-all NetworkPolicy in every namespace, with explicit allow rules for required traffic paths, limits lateral movement blast radius. If an attacker compromises a pod in the frontend namespace, a NetworkPolicy that denies egress to the database namespace except on port 5432 from the api namespace prevents that pod from directly querying databases.
NetworkPolicy is underused. A 2024 survey of K8s security posture across cloud environments found that the majority of production namespaces had no NetworkPolicy objects at all — meaning compromise of any pod in those namespaces provides unrestricted access to any other pod in the cluster network.
Layer 3: Runtime Security
Runtime security covers events that occur after a container is running: process executions, syscalls, file operations, network connections, namespace operations. This is the layer that build-time and deploy-time controls cannot cover, because the events happen in a running process, not in a configuration file or container image.
eBPF-Based Syscall Monitoring
The practical deployment is a DaemonSet running an eBPF agent on every node, capturing syscall events via kernel tracepoints and evaluating them against detection rules. The detection coverage at this layer includes: container escapes (namespace operations, capability escalation), cryptomining (process names, stratum protocol connections), lateral movement (unexpected cross-namespace connections, credential file accesses), privilege escalation (setuid/setgid calls, capability requests), and zero-day exploitation patterns (kernel exploitation post-compromise behavioral indicators).
This layer catches what build and deploy can't: a zero-day exploited post-deploy, a legitimate-but-compromised image that passed scanning, a misconfigured RBAC that lets an attacker abuse a ServiceAccount, or an insider threat running unexpected processes.
K8s Audit Log Analysis
The Kubernetes API server audit log records every API operation: create pod, delete secret, get serviceaccount/token, exec into pod. Threat patterns visible only in audit logs: service account token exfiltration, RBAC manipulation (create clusterrolebinding by a non-admin user), mass enumeration (TA0007 Discovery tactics — listing secrets across namespaces, iterating over configmaps). Correlating audit log events with process-level events from eBPF closes the gap where API-level attacks are invisible to syscall monitoring and vice versa.
Where Teams Actually Have Gaps
Running kube-bench (the CIS Kubernetes Benchmark scanner) against most mid-size clusters surfaces the typical pattern: L1 (build) checks pass at ~80%, L2 (deploy) checks pass at ~50%, L3 (runtime) checks fail at ~70%. The most commonly failing runtime checks: no audit logging enabled on the API server, no runtime threat detection agent deployed, NetworkPolicy not applied to all namespaces.
We're not saying covering only two of three layers is negligent — we're saying it represents a specific, discoverable risk surface. A well-enforced deploy layer (restricted PodSecurityStandards + NetworkPolicy everywhere) significantly narrows the runtime attack surface without requiring a runtime agent. But it doesn't eliminate it. Kernel CVEs, supply chain compromises, and zero-days operate below the layer that admission controllers can see.
Use kube-bench and kube-hunter (a penetration testing tool for K8s clusters) to audit your current posture across all three layers before deciding where to invest next. The Kubesentry platform covers Layer 3 via an eBPF DaemonSet — see how runtime detection fits alongside the build and deploy layers you likely already have in place. For specific runtime threat patterns, the container escape and privilege escalation use case pages cover the detection mechanisms in detail.