domains / families / devops / jetson-llm-serve

jetson-llm-serve

separate

Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.

When to use

• Provide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.
• "Run / serve / host this model on a Jetson."
• "Start a vLLM server I can hit from Open WebUI / my app."
• After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.

When not to use

• This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.
• Container tags such as `latest` are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.
• vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use `jetson-inference-mem-tune` when a command OOMs or memory headroom matters.
• Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor / SM 11.0 correctly.
• Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.
• On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as `gdm3` or `nvargus-daemon` requires explicit user approval.
• Host-native vLLM/SGLang on Thor should be used only when that install is already validated on the target JetPack.

Skill page

Source