domains / families / devops / gke-ai-troubleshooting-node-unresponsive-timeout

gke-ai-troubleshooting-node-unresponsive-timeout

separate

Diagnose and mitigate GKE TPU or GPU nodes stuck in NotReady / NodeStatusUnknown ("Kubelet stopped posting node status") due to host kernel panics, hardware lockups, or disabled node auto-repair. Use when nodes stop heartbeating beyond the node auto-repair threshold and pods remain stuck in Terminating. Don't use for healthy nodes, pod-only application crashes, or routine GKE upgrades.

When to use

• Diagnose GKE TPU or GPU nodes stuck in NotReady
• Diagnose NodeStatusUnknown conditions
• Investigate host kernel panics
• Investigate hardware lockups
• Check disabled node auto-repair
• Troubleshoot pods stuck in Terminating
• Verify node heartbeat timeouts

When not to use

• Healthy nodes
• Pod-only application crashes
• Routine GKE upgrades

Skill page

Source