Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Script to convert Grok-1 weights from raw JAX pickle files. #7058

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

Script to convert Grok-1 weights from raw JAX pickle files. #7058

wants to merge 13 commits into from
+467 −0

Commits on May 25, 2024

  1. Script to convert Grok-1 weights from raw JAX pickle files.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    6ddf93b View commit details
    Browse the repository at this point in the history

  2. Don't split MoE weights. 

    As per ggerganov#7058 (comment).
This helps avoid a memcopy when running.
    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    3c57743 View commit details
    Browse the repository at this point in the history

  3. Use only one list of weight names, with values from the gguf module. 

    This saves weights in the order in which they are in the Grok-1 files.
Since we operate weight-by-weight now, we no longer need caches and
name2key translations.

Per reviewer request, I also moved to using keys in gguf.TENSOR_NAMES.
    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    0842763 View commit details
    Browse the repository at this point in the history

  4. Update convert_grok.py to use logging module

    @mofosyne @heiner
    mofosyne authored and heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    5bc4f10 View commit details
    Browse the repository at this point in the history

  5. Move print to logging: Fixes.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    d894497 View commit details
    Browse the repository at this point in the history

  6. Address review comments by foldl.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    ef671c6 View commit details
    Browse the repository at this point in the history

  7. Don't multiply embeddings with embedding_multiplier_scale as it happe… 

    …ns in llama.cpp.
    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    9a0629d View commit details
    Browse the repository at this point in the history

  8. Fix layer order.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    f177b65 View commit details
    Browse the repository at this point in the history

  9. Use Q8_0 quantization from gguf module. 

    This makes tensors exactly as in https://huggingface.co/Arki05/Grok-1-GGUF/tree/main/Q8_0
    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    e2f13a3 View commit details
    Browse the repository at this point in the history

  10. More constants from gguf.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    60b29ea View commit details
    Browse the repository at this point in the history

  11. Write tensors in layer order.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    0a1ef11 View commit details
    Browse the repository at this point in the history

  12. Move noqa comment to where the lastest flake8 likes it.

    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    abc958b View commit details
    Browse the repository at this point in the history

  13. Implement Q8_0 quantization fully in PyTorch. 

    This is equivalent to gguf.quantize_q8_0 but doesn't round-trip to
Numpy.
    @heiner
    heiner committed May 25, 2024
    Configuration menu
    Copy the full SHA
    739648f View commit details
    Browse the repository at this point in the history