What started as a personal hobby project out of pure frustration with opaque Kubernetes OOMKilled events has finally graduated into a diagnostic tool that is actually pretty useful.

In Kubernetes, when a container exceeds its cgroup memory limit, the Linux kernel invokes the OOM killer. From the cluster operator’s perspective, the pod simply vanishes, and the replacement pod reports Reason: OOMKilled.

But why did it OOM? Was the application suddenly allocating a huge amount of memory? Did a sidecar container spike and cause the main app to be killed? Standard metrics only show the memory climbing up to the cliff, not what pushed it over. By the time Kubernetes notices the pod is dead, the kernel has already destroyed the cgroup, the memory footprint, and the filesystem.

Evolution of the Architecture

V1: The User-Space Polling Era (inotify & cgroups)

My first iteration attempted to solve this entirely in user-space by watching the memory.events file in cgroups using inotify.

  graph TD
    subgraph Node[Kubernetes Node]
        K[Kernel Cgroup v2] -->|memory.events| I[inotify Watcher]
        I -->|OOM Event Triggered| P[Poll /proc]
        P -.->|Too Slow!| C[Cgroup already destroyed]
    end

The problem was fundamental race conditions. By the time inotify propagated the event to the user-space agent, the OOM killer had already struck. The kernel destroys the cgroup almost instantly after the process dies. The agent would rush to read /proc/$PID/smaps or /sys/fs/cgroup/..., only to find the directory missing. It was inherently flaky.

V2: The eBPF Era (Current)

To capture the state before destruction, you have to be in the kernel. kube-autopsy now uses eBPF (Extended Berkeley Packet Filter) to attach a kprobe directly to the oom_kill_process kernel function.

This changes the paradigm from polling to synchronous interception.

  graph TD
    subgraph Kernel[Linux Kernel Space]
        OOM[oom_kill_process] -->|kprobe intercept| BPF[eBPF Program]
        BPF -->|Read mm_struct| M[Capture RSS, Pages, Victim]
        BPF -->|Write| RB[Zero-Copy Ringbuffer]
    end
    
    subgraph UserSpace[User Space / Node Agent]
        RB -->|Read| A[Agent DaemonSet]
        A -->|Tail /var/log/pods| L[Capture Last Logs]
        A -->|Format JSON| API[Kubernetes API]
    end
    
    API -->|Create| CRD[PodCrashReport CRD]
    
    subgraph Controller[Controller / Control Plane]
        CRD --> C[Manager]
        C -->|Dispatch| W[Slack Webhooks]
        C -->|Expose| Prom[Prometheus Metrics]
    end

The core of kube-autopsy is a eBPF program written in C and managed via Go’s cilium/ebpf library.

When the kernel decides to kill a process, it calls oom_kill_process. kube-autopsy with the eBPF program intercepts this function call. At this exact moment, the process is frozen in time—it hasn’t been killed yet, and its memory structures are fully intact.

  1. We identify the Trigger via bpf_get_current_pid_tgid() and it gives us the PID of the process that attempted the fatal memory allocation.
  2. We identify the Victim via oom_kill_process which contain the task_struct of the process the kernel selected to kill.
  3. We dissect the memory by walking the kernel’s task_struct -> mm_struct, and we can precisely read the anon_rss, file_rss, and page_tables_bytes footprint of the victim.
  4. We get the cgroup contex via the origin cgroup. If the OOM was triggered by the container hitting its specific memory limit, it’s a ContainerLimit OOM. If the physical node ran out of memory, it’s a NodeExhaustion OOM.

This data is packed into a struct and pushed into a BPF Ringbuffer. The user-space Go agent blocks on this ringbuffer using epoll. Because it’s a ringbuffer, the data transfer is lockless and zero-copy, adding virtually zero overhead to the kernel’s critical path.

The Kubernetes Bridge

The eBPF program operates purely on kernel constructs (PIDs, cgroup IDs, memory pages). The Go agent’s job is to bridge this to into Kubernetes.

When an event arrives via the ringbuffer, the agent:

  1. Maps the raw container ID to a running Pod using the Kubelet API.
  2. Quickly tails /var/log/pods/ to grab the final 50 lines of stdout/stderr before the runtime deletes the log file.
  3. Constructs a PodCrashReport Custom Resource Definition and pushes it to the Kubernetes API server.

By creating a CRD (autopsy.tty.se/v1alpha1), the crash data becomes a native Kubernetes object. It is tied to the crashed pod via an OwnerReference, meaning when the pod is finally garbage collected, the crash report dies with it (or persists for 24h via a custom GC if configured).

kube-autopsy tries to elimiate the race conditions that plague traditional OOM debugging. What started as a hacky inotify script is now a cluster-wide diagnostic system.