domains / families / devops / gke-ai-troubleshooting-tpu-slice-mig-stuck-updating

gke-ai-troubleshooting-tpu-slice-mig-stuck-updating

separate

Diagnose and resolve GKE TPU Dynamic Slice creation and JobSet scheduling failures blocked by underlying Compute Engine managed instance groups (MIGs) stuck in an updating or unstable state (isStable=False). Use when TPU slice creation fails with a SliceCreationFailed status reason, when JobSets remain stuck in pending, or when MIG status reflects isStable=False or versionTarget.isReached=False during TPU slice formation. Don't use for SliceCreationFailed causes unrelated to MIG stability (use gke-ai-troubleshooting-tpu-dynamic-slices-monitoring) or general pod scheduling failures unrelated to TPU slices.

When to use

• TPU slice creation fails with SliceCreationFailed status reason
• JobSets remain stuck in pending
• MIG status reflects isStable=False
• MIG status reflects versionTarget.isReached=False
• TPU slice formation blocked by unstable MIGs

When not to use

• SliceCreationFailed causes unrelated to MIG stability
• General pod scheduling failures unrelated to TPU slices

Skill page

Source