Understanding Positional Embeddings in Modern LLMs

Sebastian Raschka, PhD, published a technical walkthrough on his Ahead of AI newsletter, dated November 15, 2026, on how large language models signal token order. The original reporting appears at magazine.sebastianraschka.com. It moves from GPT-2's learned position tables to rotary position embedding (RoPE), the method now common in open models. Raschka works through GPT-2 XL dimensions and a toy example in which an eight-feature attention head splits into four feature pairs.
A learned table supplies a separate vector for each position, and the model sums it with the token embedding before the first transformer block. RoPE takes a different route: it rotates the query and key vectors inside each attention layer by an angle tied to position, so attention scores reflect distance between tokens. Standard implementations use a frequency base of 10,000. Raschka also cites a paper finding that rotating only about 10% of features matched full RoPE on training loss and benchmark accuracy, though full RoPE scored better on some evaluations.
Raschka expects the rotary method to remain the default through 2027, given how widely it is used today.
Why it matters
Position handling is an architecture choice with cost consequences. Rotary embeddings do without a fixed learned table of positions, which helps explain their spread. A company that fine-tunes or serves an open base model therefore inherits that model's positional scheme. The scheme also shapes how far a context window can be pushed and how much work that takes.
For diligence, the useful questions are which positional scheme a product relies on, what context length was actually tested, and whether any extension method was applied. Being able to compute positions beyond the training range does not by itself guarantee quality at those lengths, so window claims deserve evaluation at the stated size. Founders pitching long-context features should expect technical buyers to ask for those numbers.
Key points
- 1Older GPT-style models store one learned position vector per slot and add it to token embeddings at input.
- 2RoPE rotates query and key vectors inside each attention layer by an angle tied to token position.
- 3Standard RoPE uses a frequency base of 10,000, according to an arXiv paper on length extrapolation.
- 4Rotating about 10% of features matched full RoPE on training loss and benchmark accuracy in one cited study.
- 5Raschka forecasts that RoPE stays the default positional method through 2027.