domains / families / devops / jetson-llm-benchmark

jetson-llm-benchmark

separate

Benchmark Jetson LLM/VLM serving performance across vLLM, llama.cpp, and Ollama with structured JSON output.

When to use

• Measure deployed LLM latency and throughput on a Jetson target using the correct runtime-specific benchmark wrapper. Use the JSON output to compare models, runtime flags, power modes, and before/after tuning changes.
• "Benchmark / measure / compare X on this Jetson."
• After `jetson-llm-serve` to actually quantify the deployment.
• Before/after applying flags from `jetson-inference-mem-tune` to confirm the change helped.

When not to use

• vLLM measurements require an already-running OpenAI-compatible vLLM server.
• Ollama results are single-stream by design and are not directly comparable to
• llama.cpp/GGUF benchmarking runs a NVIDIA-AI-IOT container by default. Tell the
• Container image tags may be mutable unless the caller passes a digest-pinned
• Results are only comparable when model, quantization, prompt length, output

Skill page

Source