domains / families / devops / dynamo-interconnect-check

dynamo-interconnect-check

separate

Validate that a Dynamo deployment's NIXL/UCX/NCCL interconnect is ready for disaggregated serving over RDMA/NVLink. Use after recipe-runner brings a deployment up (especially disagg/multi-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.

When to use

• Confirm that the transport disaggregated serving depends on actually works. A deployment can pass an endpoint smoke test while disagg is silently wrong: if NIXL/UCX cannot reach the peer worker over RDMA or NVLink, KV transfer falls back t…
• After `dynamo-recipe-runner` deploys a **disagg** or multi-node recipe.
• Before reporting disagg throughput/latency, so numbers reflect the real
• When agg works but disagg is slow, hangs, or returns wrong output and you

When not to use

• Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools).
• `skipped` results for missing tools (`ibstat`, `nvidia-smi`, `lsmod`) are inconclusive, not a pass.
• Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected.
• Single-node agg deployments do not exercise the transport — this skill is for disagg / multi-node validation.

Skill page

Source