length-extrapolation

📰 Must-read papers and blogs on LLM based Long Context Modeling 🔥

Methods and code for extending the context length of diffusion language models

An imaginary extension of rotary position embeddings for long-context language models

Context-aware Biases for Length Extrapolation

A training-free method for extending the maximum context length of language models

Subquadratic sparse attention that zero-shot length extrapolation to 256k+ tokens, with fast CUDA kernels. arXiv:2610.17191

Controlled experiments on recurrent-depth (looped) transformers — length extrapolation, adaptive test-time compute, and honest sub-1B negatives.

Selective Attention trained like Dilution-Attention: no position signal, 209M parameters, 16K context, with a fused Triton kernel.

Likelihood-based Needle-In-A-Haystack eval for small and from-scratch language models: multiple-choice scoring with a control prior (no generation needed), a hard mode with natural filler and keyed distractors, depth × length heatmaps, during-training eval and an LM Evaluation Harness task.

A modular, config-driven PyTorch framework for conducting empirical positional encoding ablation experiments on decoder-only transformers (~51M parameters) using Google Colab T4 GPUs

A constant-memory recurrent stream coupled to a growing .causal knowledge index, with a measured capacity threshold in the gated readout. Builds on GSSM.

The O(1)-state organism: constant-memory unbounded streaming, surprise-gated plasticity, holographic memory through silence, dosed sleep, a runtime knowledge index — and live cross-architecture migration. Seven disclosed foundations, 38 pre-registered predictions, falsifications kept.