domains / families / research / long-context

long-context

separate

Extend context windows of transformer models using RoPE, YaRN, ALiBi, and position interpolation techniques. Use when processing long documents (32k-128k+ tokens), extending pre-trained models beyond original context limits, or implementing efficient positional encodings. Covers rotary embeddings, attention biases, interpolation methods, and extrapolation strategies for LLMs.

When to use

• **Process long documents** (32k, 64k, 128k+ tokens) with transformer models
• **Extend context windows** of pre-trained models (LLaMA, Mistral, etc.)
• **Implement efficient positional encodings** (RoPE, ALiBi)
• **Train models** with length extrapolation capabilities
• **Deploy models** that handle variable-length inputs efficiently
• **Fine-tune** existing models for longer contexts with minimal compute

Skill page

Source