domains / families / research / rwkv

rwkv

separate

RNN+Transformer hybrid with O(n) inference. Linear time, infinite context, no KV cache. Train like GPT (parallel), infer like RNN (sequential). Linux Foundation AI project. Production at Windows, Office, NeMo. RWKV-7 (March 2025). Models up to 14B parameters.

When to use

• Need very long context (100K+ tokens)
• Want constant memory usage
• Building streaming applications
• Need RNN efficiency with Transformer performance
• Memory-constrained deployment
• **Linear time**: O(n) vs O(n²) for Transformers
• **No KV cache**: Constant memory per token
• **Infinite context**: No fixed window limit

When not to use

• Tasks requiring absolute best performance with available compute
• State-space models (Mamba)
• Retention mechanism (RetNet)
• Convolution-based approach (Hyena)

Skill page

Source