domains / families / devops / prefix-cache-replay

prefix-cache-replay

separate

Replay an LLM inference request trace (Mooncake / vLLM / SGLang hash_ids format) against a block-level KV prefix cache and compute hit statistics. Use when given a request trace plus cache configuration and asked for hit rate, hit tokens, or final cache contents. Covers the longest-contiguous-prefix semantics that distinguishes KV prefix caching from full-prompt prompt caching, the policy-specific residency and eviction rules (LRU, LFU, S3FIFO), and the partial-last-block accounting rule.

When to use

• Replay LLM inference request trace
• Compute hit rate
• Compute hit tokens
• Compute final cache contents
• Handle longest-contiguous-prefix semantics
• Apply LRU eviction policy
• Apply LFU eviction policy
• Apply S3FIFO eviction policy

When not to use

• Full-prompt prompt caching

Skill page

Source