Beyond Access Logs: Using Kubernetes Audit Events for Runtime Anomaly Detection

What the K8s Audit Log Actually Contains

The Kubernetes API server audit log records every operation that passes through the API: creates, reads, updates, deletes, watches, execs, and port-forwards — tagged with the requesting user (or ServiceAccount), the verb, the resource, the namespace, the response code, and a timestamp. It's a high-fidelity record of control plane activity, and it captures threat patterns that syscall-level monitoring cannot see.

An attacker who achieves code execution in a pod and uses the pod's ServiceAccount token to call the K8s API — creating new pods, listing secrets, binding a ClusterRole to their ServiceAccount — does so entirely at the API level. The API calls appear in the audit log as normal authenticated requests. The eBPF agent on the node sees the network connection going out to the API server but cannot interpret the HTTPS payload. The audit log is where those API-layer attacks are visible.

Enabling API server auditing requires explicit configuration. EKS, GKE, and AKS each have their own controls for audit log enablement and destination — typically CloudWatch Logs, Google Cloud Logging, or Azure Monitor respectively. Self-managed clusters require configuring the --audit-log-path and --audit-policy-file flags on the kube-apiserver. The audit policy file controls which events are logged at what detail level; a minimal policy that logs all Metadata-level events plus Request-level for sensitive resources (secrets, configmaps, serviceaccounts/token) provides a good coverage-to-volume tradeoff.

The Four Attack Patterns Visible Only in Audit Logs

ServiceAccount Token Exfiltration

The Kubernetes API endpoint POST /api/v1/namespaces/{namespace}/serviceaccounts/{name}/token generates a bound token for a ServiceAccount. An attacker with sufficient RBAC permissions calling this endpoint can obtain tokens for ServiceAccounts more privileged than their current access — a form of privilege escalation that happens entirely through API calls, not syscalls. The audit log records the request, the requesting identity, and the target ServiceAccount. The detection pattern: a ServiceAccount or user requesting tokens for ServiceAccounts in namespaces other than their own, or requesting tokens for ServiceAccounts with cluster-admin-adjacent permissions, when the requesting identity has no legitimate business reason for those requests.

RBAC Manipulation

Creating a ClusterRoleBinding that grants cluster-admin privileges to a compromised ServiceAccount is a classic persistence technique following initial access. The API call is: POST /apis/rbac.authorization.k8s.io/v1/clusterrolebindings with a body binding the cluster-admin ClusterRole to the attacker's ServiceAccount. This is entirely an API operation — no syscalls involved, no process created, no file written on any node. It's invisible to eBPF syscall monitoring. The audit log records it: requestURI, verb=create, resource=clusterrolebindings, the full request object body, and the requesting user. A non-admin user (or any ServiceAccount) creating ClusterRoleBindings to high-privilege roles is unambiguously anomalous.

Mass Enumeration (MITRE TA0007 Discovery)

Before exfiltrating secrets or credentials, attackers typically enumerate what's available. A K8s cluster with secrets scattered across namespaces requires first knowing what exists. The MITRE ATT&CK container tactics TA0007 (Discovery) patterns include: listing all secrets across all namespaces (list /api/v1/secrets), listing all configmaps (list /api/v1/configmaps), listing all pods (list /api/v1/pods with namespace=all), and listing all ClusterRoles and ClusterRoleBindings to understand privilege relationships. These requests appear legitimate individually — a ServiceAccount might legitimately list secrets in its own namespace — but the combination, velocity, and scope of a discovery sweep is distinguishable from normal operational patterns. A single ServiceAccount listing secrets across 15 namespaces in under 30 seconds is not normal CI/CD behavior.

Exec and Port-Forward Abuse

The kubectl exec capability maps to the API endpoint POST /api/v1/namespaces/{namespace}/pods/{name}/exec with upgrade to websocket. An attacker with pods/exec RBAC permission (often granted to developers for debugging) who has compromised a user credential can exec into pods they control or pivot to adjacent pods. The audit log records the requesting user, the target pod, and the command. Unusual patterns: a user execing into pods across multiple namespaces that aren't in their team's scope, or exec sessions at unusual hours with no corresponding CI/CD pipeline activity.

The Correlation Problem: Why Neither Source Alone Is Enough

Consider a realistic attack scenario from an internal red-team simulation on a test cluster: the attacker has achieved code execution in a pod via a web application vulnerability (SSRF → IMDS endpoint on EC2 → instance role credentials → S3 read → deploy credentials → new pod). From the compromised pod, they read the mounted ServiceAccount token (/run/secrets/kubernetes.io/serviceaccount/token), query the K8s API to discover what permissions the token has, find the token has pods/exec in the production namespace, exec into a pod running in that namespace that has a more privileged ServiceAccount mounted, read that ServiceAccount's token, and use it to create a ClusterRoleBinding granting themselves cluster-admin.

What each monitoring source sees in isolation:

• eBPF syscall monitor: sees the open() call reading /run/secrets/kubernetes.io/serviceaccount/token (anomalous — this file read outside of expected init path). Also sees the outbound TCP connections to the K8s API server. Does not see the HTTPS content of those connections.
• K8s audit log alone: sees ServiceAccount A listing pods in production namespace, then an exec into pod X, then ServiceAccount B (from pod X) listing ClusterRoles, then ServiceAccount B creating a ClusterRoleBinding to cluster-admin. Individually, some of these steps look unusual but could have benign explanations. The CreateClusterRoleBinding is clearly anomalous.
• Both sources correlated: the ServiceAccount token read event from eBPF and the API call from that ServiceAccount happening within seconds of each other ties the syscall event to the API action. The pivot to ServiceAccount B in pod X, followed by the API calls from B, becomes a connected chain rather than isolated anomalies. The correlation reduces investigation time from "something weird happened in audit logs" to "this pod on this node read its token, then used it to execute into this other pod, which then escalated to cluster-admin — here's the full chain."

Setting Up Audit Log Forwarding and Analysis

A practical audit policy for threat detection purposes (not compliance theater) focuses on the high-value event types:

apiVersion: audit.k8s.io/v1
kind: Policy
rules:
# Log all token request operations at RequestResponse level
- level: RequestResponse
  verbs: ["create"]
  resources:
  - group: ""
    resources: ["serviceaccounts/token"]

# Log RBAC mutations at RequestResponse level
- level: RequestResponse
  verbs: ["create", "update", "patch", "delete"]
  resources:
  - group: "rbac.authorization.k8s.io"
    resources: ["clusterrolebindings", "rolebindings", "clusterroles"]

# Log exec and port-forward at Request level (websocket upgrade)
- level: Request
  verbs: ["create"]
  resources:
  - group: ""
    resources: ["pods/exec", "pods/portforward"]

# Log secret access at Metadata level (don't log secret values)
- level: Metadata
  verbs: ["get", "list", "watch"]
  resources:
  - group: ""
    resources: ["secrets", "configmaps"]

# Log everything else at Metadata level
- level: Metadata

Note the Metadata level for secret access — this records that secrets were accessed but not their content. Logging secret values at RequestResponse level would create an audit log that is itself a high-value credential target. Metadata gives you the access pattern without the data.

Forwarding this audit stream to a SIEM or the Kubesentry event correlation engine enables the cross-source detection that neither the audit log nor the eBPF stream provides in isolation. The correlation window — events from both sources within a configurable time window sharing a container ID or user identity — is where multi-step attack chains become visible as chains rather than disconnected anomalies.

Limits: What Audit Logs Don't Show

We're not saying K8s audit logs are comprehensive — they're the API control plane view, not the data plane view. Everything that happens inside a container that doesn't involve a K8s API call is invisible to audit logs. A cryptomining process running entirely within a pod's compute resources, making only outbound network connections to a mining pool, generates no K8s API events at all. Container escapes that happen at the kernel level without K8s API calls are not in the audit log. Lateral movement via container network connections (pod-to-pod on the same node, bypassing the API server) is also absent.

This is why the correlation architecture matters: audit logs cover API-plane attacks, eBPF syscall monitoring covers execution-plane attacks, and the overlap between them — API calls correlated with the container events that triggered them — is where the highest-confidence detections live. See the Kubesentry platform for how audit log analysis integrates with the eBPF event stream, or read the three-layer K8s security model for where audit log analysis fits in the broader security coverage map. For the specific detection rules around RBAC manipulation and ServiceAccount abuse, the detection documentation covers the full rule set.