Llama 3 achieves pretty good recall to 65k context w/ rope_theta set to 16M

[dranger003]

Source: https://twitter.com/winglian/status/1783122644579090600

I'm not sure how this can be applied using llama.cpp but when I try with -c 32768 --rope-scaling linear --rope-freq-base 8000000 I get coherent and high quality results from the model.

Am I using the right parameters? I also noticed the VRAM usage doesn't go up all that much and I can easily run the Q8_0 of the 8B version on 24GB (it uses only 16.5GB fully offloaded). Actually, I can even run the FP16 fully offloaded using 23GB and 32K context.

The performance is also quite acceptable (this is on 4090):

llama_print_timings:        load time =    3333.42 ms
llama_print_timings:      sample time =      36.12 ms /   487 runs   (    0.07 ms per token, 13483.95 tokens per second)
llama_print_timings: prompt eval time =   12897.15 ms / 24642 tokens (    0.52 ms per token,  1910.65 tokens per second)
llama_print_timings:        eval time =   10108.76 ms /   486 runs   (   20.80 ms per token,    48.08 tokens per second)
llama_print_timings:       total time =   23296.00 ms / 25128 tokens

[disperaller]

does anyone know why setting the rope theta to 16m in order to achieve 65k context length according to the twitter source? in my understanding, if we want to extend the context length from 8k (llama3 8b supports naturally) to 65k, we need a multiply factor of 8. Based on the original setting of llama3 8b, the rope theta was set to 0.5m, and we multiply that by 8, we get 4m instead of 16m. Can't figure out where the difference comes from. Thanks in advance.

[dranger003]

The answer may lie these posts:

https://blog.eleuther.ai/rotary-embeddings/
https://blog.eleuther.ai/yarn/