The increased base RoPE value from Llama2 to Llama3

[FlyingFish760]

In the text part of "1.2 Modified RoPE" in the file "converting-llama2-to-llama3.ipynb". I am a little confused about the description "Increasing the base from 10,000 to 500,000 makes the frequencies (or rotation angles) decay more slowly across the dimensions, which means that higher dimensions will be associated with larger angles than before". But according to the formula of the frequency:
$${ \theta_i = \text{base}^{\frac{-2(i-1)}{d}}, i \in {1, 2, \dots, \frac{d}{2}} }$$
Should it not be that the frequency decreases quicker with a larger base value due to the negative exponent? (Imagine $0.1^i$ and $0.01^i$)

[casinca]

Hello,

From my perspective re-reading, I'd say you are correct, It seems like Sebastian inverted.
That would be something like: a larger base means frequencies decrease faster and therefore angles/rotations are getting smaller/slower across dimensions (vs a smaller base, for the same position)

This needs to be double checked but that would make sense when larger LLMs tend to increase the base to better adapt for longer contexts and minimize overlaps/increase slowdown.

Edit: a little desmos is even better to visualize

[d-kleine]

@rasbt What do you think?

[rasbt]

Thanks for the feedback @FlyingFish760 @casinca (and thanks for the reminder @d-kleine).

Looks like I made a mistake there 😅. Thanks for calling that out. Fixing it via #910

[d-kleine]

For those who are curious, I found an excellent resource that visualizes how in RoPE higher dimensions are associated with smaller rotation angles:
https://amaarora.github.io/posts/2025-09-21-rope-context-extension.html#intuition-behind-rope

(Unfortunately, there is no option to modify the theta value)

[rasbt]

Looks like a great resource!