domains / families / devops / jetson-inference-mem-tune

jetson-inference-mem-tune

separate

Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM/VLM workload on any NVIDIA Jetson.

When to use

• Turn a live `jetson-memory-audit` snapshot into runtime and launch-flag recommendations for LLM/VLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.
• "Which serving stack should I use on Orin Nano 8 GB to run a 7B model?"
• "vLLM is OOMing — what should `--gpu-memory-utilization` and `--max-model-len` be?"
• "Same model, less memory — can I switch from vLLM to llama.cpp?"
• After `jetson-memory-audit` shows a model server is the top NvMap / PSS consumer.

When not to use

• Recommendations are only as fresh as the audit JSON. Re-run `jetson-memory-audit` after stopping services, changing power mode, or restarting model servers.
• The script estimates memory pressure from SKU defaults and audit totals; model-specific KV-cache, quantization, and tokenizer behavior can still require benchmarking.
• This skill emits flags only. It does not start, stop, or restart model servers.

Skill page

Source