# Is LLaMA rotary embedding implementation correct?

**URL:** <https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509>\
**Category:** 🤗Transformers\
**Created:** [June 25, 2023, 7:35pm UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509 "2023-06-25T19:35:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![reminisce](https://avatars.discourse-cdn.com/v4/letter/r/b77776/32.png) [@reminisce](https://discuss.huggingface.co/u/reminisce)\
**Post date:** [June 25, 2023, 7:35pm UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/1 "2023-06-25T19:35:56Z")

</div>

In the LLaMA model of the Transformers library, applying rotary embedding seems not correctly implemented, specifically in the `rotate_half` function (first link). For a query vector `[1, 2, 3, 4, 5, 6]`, the expected output is `[-2, 1, -4, 3, -6, 5]`, but the function returns `[-4, -5, -6, 1, 2, 3]`. Slicing should have been interleaved. The RoFormer implementation seems correct though (second link).

> <https://github.com/huggingface/transformers/blob/8e164c5400b7b413c7b8fb32e35132001effc970/src/transformers/models/llama/modeling_llama.py#L124-L128>

> <https://github.com/huggingface/transformers/blob/8e164c5400b7b413c7b8fb32e35132001effc970/src/transformers/models/roformer/modeling_roformer.py#L328-L331>

---

<div class="post-metadata">

**Author:** ![GalacticKip7](https://avatars.discourse-cdn.com/v4/letter/g/ce7236/32.png) [@GalacticKip7](https://discuss.huggingface.co/u/GalacticKip7)\
**Post date:** [September 7, 2023, 12:32am UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/2 "2023-09-07T00:32:05Z")

</div>

@reminisce I think you’re correct. I saw the same thing. The embeddings are supposed to be interweaved. In this manner, the embeddings have an identical form at dim 0 that they would at dim x.shape[-1]//2.  
(screenshot below shows what I’m referring to in poor detail - it shows dummy data (`torch.arange(0, 256).unsqueeze(0).repeat(16, 1)` which is why there’s the smooth background color change from 0 to 256) that is encoded with this rotary PE code)

(Note: I haven’t rigorously examined this test, but this simple examination raises my concern that it’s not correct)

 ![Screenshot 2023-09-06 at 5.15.25 PM](https://us1.discourse-cdn.com/hellohellohello/original/3X/c/e/ce070b59e8e38a36c7dfe60a5709ba8d8284295a.png)

---

<div class="post-metadata">

**Author:** ![reminisce](https://avatars.discourse-cdn.com/v4/letter/r/b77776/32.png) [@reminisce](https://discuss.huggingface.co/u/reminisce)\
**Post date:** [September 7, 2023, 2:25am UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/3 "2023-09-07T02:25:31Z")

</div>

Thanks for checking this @GalacticKip7. I actually later found that simply rotating half is also a correct form of rotary embedding (see the following vector-vector multiplication-addition form and the equivalent matrix-vector multiplication form). As long as the matrix (R) satisfies the third equation for any q and k vectors, it’s a valid form of rotary embedding. The only caveat is use the same form for both training and inference.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/c/bca5a1cc61b273a78b3934357a765c427e6ee2ca.png)

---

<div class="post-metadata">

**Author:** ![GalacticKip7](https://avatars.discourse-cdn.com/v4/letter/g/ce7236/32.png) [@GalacticKip7](https://discuss.huggingface.co/u/GalacticKip7)\
**Post date:** [September 7, 2023, 4:53pm UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/4 "2023-09-07T16:53:45Z")

</div>

@reminisce Thank you for the clarification. I’ve only seen the original implementation (see pic) - your pictures help clarify this alternative derivation, though. I’ll try to go over it again once I have a pencil and paper in front of me 🙂

 ![Screenshot 2023-09-07 at 9.34.20 AM](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/e/2e40c4352d9ecefaf74c71b95b8ddf47eef0ab9a.png)

---

<div class="post-metadata">

**Author:** ![alexchen4ai](https://avatars.discourse-cdn.com/v4/letter/a/8e8cbc/32.png) [@alexchen4ai](https://discuss.huggingface.co/u/alexchen4ai)\
**Post date:** [February 6, 2024, 9:41am UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/6 "2024-02-06T09:41:28Z")

</div>

Hi I have the same question after viewing the source code in the huggingface. The key problem here is that we load the original meta llama weights to the huggingface llama model. Thus, the rotary embedding should be designed as the same structure. The source llama meta also use the same rotary embedding (let me know if I am wrong) as original rope paper.

To test if we really need the same setup during training and inference, I change the source code of transformers current main branch to use the original paper. And I get the response of this CodeLlama-7b-hf:

```auto
import socket

def ping_exponential_backoff(host: str):
    for i in range(1, 10):
    try:

    # s = socket.socket(socket.AF_INETH, socket.SOCK_STREQ, socket.SOCK_STREUSE_TCP_NODELAY)
    s.connect((host, 80))
    s.close()
    return True
  except:
    return False

def ping_host(host: str):
  for i in range(1, 10):
    if ping_host(host):
      return True
    return False

def ping_host(host: str):
  for i in range(1, 10):
    if ping_host(host):
      return False

def ping_host

```

If I keep the current transfomers main branch, I can still get meaningful result.

```auto
def ping_exponential_backoff(host: str):
    """
    Ping a host with exponential backoff.
    """
    for i in range(1, 10):
        try:
            socket.create_connection((host, 80), 5).close()
            return True
        except OSError as e:
            if i < 10:
                time.sleep(2 ** i)
            else:
                raise e

def ping_exponential_backoff_with_timeout(host: str, timeout: int):
    """
    Ping a host with exponential backoff and timeout.
    """
    for i in range(1, 10):
        try:
            socket.create_connection((host, 80), timeout).close()
            return True
        except OSError as e

```

I guess a more in-depth investigation may be required to compare the comprehensive performance change…

---

<div class="post-metadata">

**Author:** ![ShoufaChen](https://avatars.discourse-cdn.com/v4/letter/s/13edae/32.png) [@ShoufaChen](https://discuss.huggingface.co/u/ShoufaChen)\
**Post date:** [April 10, 2024, 3:29am UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/7 "2024-04-10T03:29:16Z")

</div>

Hello, @alexchen4ai

In fact the checkpoint is not totally same.  
Please see [[LLaMA] Rotary positional embedding differs with official implementation · Issue #25199 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/issues/25199#issuecomment-1687720247)

They did some weight permutation when convert llama weight to huggingface.

---

<div class="post-metadata">

**Author:** ![matkad](https://avatars.discourse-cdn.com/v4/letter/m/b77776/32.png) [@matkad](https://discuss.huggingface.co/u/matkad)\
**Post date:** [November 23, 2024, 2:17pm UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/8 "2024-11-23T14:17:53Z")

</div>

Did you check if it’s really equivalent ? I am having my doubts implementing this in Julia and I would prefer simper concatenation but I am also puzzled by two distinct PyTorch implementations.

---

<div class="post-metadata">

**Author:** ![cc007](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/cc007/32/43344_2.png) [@cc007](https://discuss.huggingface.co/u/cc007)\
**Post date:** [February 1, 2025, 1:06pm UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/9 "2025-02-01T13:06:15Z")

</div>

I am afraid your answer is not right. Since the \theta is not map correctly. And the key point why transformer’s RoPE only use rotate\_half is that in `transformers/models/llama/convert_llama_weights_to_hf.py`, there exists `permute` function, which makes the weight rearrange along with last dimension, leading like this:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/2/d/2d65e71d744297de1dee6e920d2ca99c99631a2f.png)

---

<div class="post-metadata">

**Author:** ![dev-grn](https://avatars.discourse-cdn.com/v4/letter/d/f475e1/32.png) [@dev-grn](https://discuss.huggingface.co/u/dev-grn)\
**Post date:** [February 26, 2026, 8:42am UTC](https://discuss.huggingface.co/t/is-llama-rotary-embedding-implementation-correct/44509/10 "2026-02-26T08:42:18Z")

</div>

@cc007 No. The permutation is done since the model architecture for hugging face is different. The RoPE implementation by @reminisce is also correct as well as the one inside the literature. Different implementations but they lead to the same results as long as Q, K stays untouched. If you look inside the conversion script, Q, K are permuted to match the implementation by HF.
