domains / families / data / training-check

training-check

authentic

Periodically check WandB metrics during training to catch problems early (NaN, loss divergence, idle GPUs). Avoids wasting GPU hours on broken runs. Use when training is running and you want automated health checks.

When to use

• After training is confirmed running (session alive, loss decreasing for first few steps)
• Set up via CronCreate to fire periodically during training
• **This skill checks training QUALITY, not process HEALTH.** Process health (session alive, GPU utilization) is [watchdog.py](../../tools/watchdog.py)'s job.

When not to use

• checking process health
• checking session alive status
• checking GPU utilization
• catching crashes immediately
• catching idle GPUs immediately

Skill page

Source