domains / families / devops / gke-ai-troubleshooting-tpu-performance-degradation

gke-ai-troubleshooting-tpu-performance-degradation

separate

Diagnose GKE Cloud TPU training throughput drops and step-time regressions (15%+ TPU duty-cycle drop) using ML Diagnostics Workload Monitoring (`gcloud alpha mldiagnostics monitored-events` / `hypercomputecluster.googleapis.com/v1alpha`) and 1-minute Cloud Monitoring system metrics (`kubernetes.io/node/accelerator/*`). Distinguishes hardware and network fabric throttling from workload resource bottlenecks (HBM capacity, host memory, or host CPU saturation). Use when TPU training throughput or duty cycle drops without crashing pods, when `PERFORMANCE_DEGRADATION` monitored events fire, or when triaging slow multi-slice training steps. Don't use for complete multi-slice XLA execution stalls with `HANG_DETECTED` logs (use gke-ai-troubleshooting-tpu-mxla-hang) or pod eviction/interruption restarts (use gke-ai-troubleshooting-jobset-interruption).

When to use

• TPU training throughput drops
• step-time regressions
• 15%+ TPU duty-cycle drop
• PERFORMANCE_DEGRADATION monitored events
• triaging slow multi-slice training steps
• hardware and network fabric throttling
• workload resource bottlenecks
• HBM capacity, host memory, or host CPU saturation

When not to use

• complete multi-slice XLA execution stalls
• HANG_DETECTED logs
• pod eviction/interruption restarts

Skill page

Source