Nice to see the LLaMA* idea implemented in these models:

Not sure if anyone is working on this yet but i'm happy to pick it up

same for me as well, I can pick it up. I am trying to upload the model on ollama, but got NotImplementedError: Architecture 'OpenELMForCausalLM' not supported!

Okay update: It's more difficult as I first expected it and this is a new codebase so it's stumped me a bit.
OpenELM have defined the variables: a_min, a_max for scaling the attention heads and b_min and b_max for scaling the ffn. This is both captured in their config but you can also derive it from the equation:

Still attempting it but i don't think i'm gonna be done any time soon

@joshcarp
Waiting patiently to try it out hopefully 🤞
Are you having any troubles figuring it out?

Yeah, so basically, i think I can't figure out how to calculate the kqv offsets for the kqv tensor every layer.
I'd be lying if i didn't admit i've just been throwing spaghetti at the wall as shown my the git history on this: https://github.com/joshcarp/llama.cpp

I'm using this as a reference: https://huggingface.co/apple/OpenELM-270M/blob/main/modeling_openelm.py
https://github.com/apple/corenet/tree/main/mlx_examples/open_elm

If anyone else wants to implement this feel free

If anyone can help out: #6986

Does this work? Did anyone try this?

https://huggingface.co/LiteLLMs/OpenELM-GGUF

Does this work? Did anyone try this?

https://huggingface.co/LiteLLMs/OpenELM-GGUF

Have you checked the files?
It specifically states that it is just a placeholder, there is no model

Is it possible to have support forr openelm
I am curious for this because this can be run on mobile devices with less power don't know about the performance though. Compare to gemma:2b or phi3