domains / families / devops / gke-ai-troubleshooting-tpu-vbar-oom

gke-ai-troubleshooting-tpu-vbar-oom

separate

Diagnoses and prevents vbar_control_agent segfaults, out-of-memory (OOM) errors, and TPU device initialization failures on TPU v6e nodes in GKE caused by race conditions during TPU device resets or high-frequency metrics polling. Use when troubleshooting vbar_control_agent crashes, memory cgroup OOMs in serial console logs, tpu-device-plugin metrics checksum corruption errors, or custom TPU metrics collection conflicts on GKE TPU v6e nodes. Don't use for general non-TPU container OOM troubleshooting or standard GKE node lifecycle operations.

When to use

• troubleshooting vbar_control_agent crashes
• memory cgroup OOMs in serial console logs
• tpu-device-plugin metrics checksum corruption errors
• custom TPU metrics collection conflicts on GKE TPU v6e nodes

When not to use

• general non-TPU container OOM troubleshooting
• standard GKE node lifecycle operations

Skill page

Source