Send Kubernetes audit logs

Kubernetes audit logs record every request to the Kubernetes API server: who made it, which verb and resource it touched, and whether it was allowed. They show who exec’d into a pod, read a secret, or changed a role binding. There is no separate Kubernetes log source: audit logs arrive through the log source your platform already uses. On Amazon EKS that is AWS CloudWatch, on Google Kubernetes Engine (GKE) it is Google Cloud, and on a self-managed cluster it is the OpenTelemetry Collector. Pick the section for your cluster.

Send EKS audit logs

Recommended when: your cluster runs on Amazon EKS. EKS writes control-plane logs to the CloudWatch log group /aws/eks/<cluster name>/cluster. Audit events are in the log streams named kube-apiserver-audit-<id>.

Turn on the audit control-plane log type for the cluster. In the AWS console, open the cluster’s Observability tab and choose Manage logging under Control plane logging, or run:

aws eks update-cluster-config \
  --region us-east-1 \
  --name my-cluster \
  --logging '{"clusterLogging":[{"types":["audit"],"enabled":true}]}'

Replace the region and cluster name with your own. Control-plane log types you already turned on stay on.

Connect the AWS CloudWatch log source to the cluster’s AWS account, in a Sazabi project in the cluster’s region. To limit Sazabi’s role to EKS log groups when you create it with CloudFormation, set the stack’s AllowedLogGroupPrefix parameter to /aws/eks/.

Open the CloudWatch log source in Settings > Log streams, click Add log stream, and select /aws/eks/<cluster name>/cluster.

CloudWatch allows at most two subscription filters per log group, and Sazabi uses one. AWS bills CloudWatch ingestion and storage for control-plane logs; audit is the highest-volume type. AWS sets the EKS audit policy, so you cannot change which requests it records.

Send GKE audit logs

Recommended when: your cluster runs on Google Kubernetes Engine. GKE writes Kubernetes audit logs to Cloud Audit Logs in the cluster’s project, with the service name k8s.io and the resource type k8s_cluster. Admin Activity audit logs (writes such as creating a pod, pods/exec, and RBAC changes) are always on. Data Access audit logs (reads such as get secrets) are off by default.

Connect the Google Cloud log source and add a log stream for the cluster’s GCP project. The sink Sazabi creates forwards every log entry in the project, Cloud Audit Logs included, so Admin Activity audit logs arrive with no extra setup.If you send to an endpoint instead, use this filter on the Google Cloud (send to an endpoint) sink to forward only Kubernetes audit logs:

logName:"cloudaudit.googleapis.com" AND resource.type="k8s_cluster"

Optional: to record reads, turn on Data Access audit logs. In the Google Cloud console, open IAM & Admin > Audit Logs, select Kubernetes Engine API, check Data Read and Data Write, and save. Data Access logs are high volume, and Cloud Logging and Pub/Sub bill for them.

Send self-managed cluster audit logs

Recommended when: you run the Kubernetes control plane yourself (kubeadm, k3s, or similar). The API server writes audit events as JSON lines to a file, and an OpenTelemetry Collector on each control-plane node reads that file and sends it to Sazabi.

Write an audit policy. This one records request metadata for every request, which covers exec, secret reads, and RBAC changes without storing request bodies:

/etc/kubernetes/audit-policy.yaml

apiVersion: audit.k8s.io/v1
kind: Policy
omitStages:
  - RequestReceived
rules:
  - level: Metadata

Start kube-apiserver with the log backend turned on:

--audit-policy-file=/etc/kubernetes/audit-policy.yaml
--audit-log-path=/var/log/kubernetes/audit/audit.log
--audit-log-maxage=7
--audit-log-maxbackup=5
--audit-log-maxsize=100

If the API server runs as a static pod, mount the policy file and the log directory into it.

Add the OpenTelemetry Collector log source and copy its Intake URL. Run a Collector on each control-plane node that reads the audit log:

otelcol.yaml

receivers:
  filelog/k8s-audit:
    include: [/var/log/kubernetes/audit/*.log]
    start_at: end

exporters:
  otlphttp/sazabi:
    endpoint: <your intake URL>

service:
  pipelines:
    logs:
      receivers: [filelog/k8s-audit]
      exporters: [otlphttp/sazabi]

Replace <your intake URL> with the intake URL you copied. Fluent Bit works too: point a tail input at the same path and send to the Fluent Bit log source.

Verify

Confirm an audit event reaches your project.

Make a write request against the cluster, which every audit setup above records (GKE Admin Activity logs included): kubectl create configmap sazabi-audit-check -n default. Delete it afterwards with kubectl delete configmap sazabi-audit-check -n default.
In a thread, ask the Sazabi agent for Kubernetes audit events from the last 15 minutes, or run sazabi logs ask "kubernetes audit events that created the sazabi-audit-check configmap in the last 15 minutes".
You should see an event with verb create, resource configmaps, your user name, and response code 201. On GKE the method is io.k8s.core.v1.configmaps.create. EKS and GKE deliver audit events within a few minutes; a self-managed Collector delivers them within a minute.

If nothing appears, confirm the audit log type is on (EKS), the log stream for the cluster’s project exists (GKE), or the Collector can read the audit log file (self-managed).

AWS CloudWatch

Connect your AWS account and pick log groups.

Google Cloud

Connect your Google Cloud account and pick projects.

Overview

Get started

Guides

Catalogs

Reference

Security & trust

cve.now

Send Kubernetes audit logs

Send EKS audit logs

Send GKE audit logs

Send self-managed cluster audit logs

Verify

Further reading

AWS CloudWatch

Google Cloud

​Send EKS audit logs

​Send GKE audit logs

​Send self-managed cluster audit logs

​Verify

​Further reading

AWS CloudWatch

Google Cloud

Send EKS audit logs

Send GKE audit logs

Send self-managed cluster audit logs

Verify

Further reading