Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decide pre tokenizer based on preprocessing of entry and not on tokens encoded #7039

Closed
wants to merge 6 commits into from
Closed

Decide pre tokenizer based on preprocessing of entry and not on tokens encoded #7039

wants to merge 6 commits into from

Commits on May 2, 2024

  1. feat: change convert hf to gguf

    @JoanFM
    JoanFM committed May 2, 2024
    Configuration menu
    Copy the full SHA
    a1aa65e View commit details
    Browse the repository at this point in the history

  2. use conver ids to tokens

    @JoanFM
    JoanFM committed May 2, 2024
    Configuration menu
    Copy the full SHA
    0672cd8 View commit details
    Browse the repository at this point in the history

  3. fix: only do pre tokenization and normalization

    @JoanFM
    JoanFM committed May 2, 2024
    Configuration menu
    Copy the full SHA
    0f94ff7 View commit details
    Browse the repository at this point in the history

  4. fix: protect against slow tokenizer

    @JoanFM
    JoanFM committed May 2, 2024
    Configuration menu
    Copy the full SHA
    c7a6c32 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2024

  1. fix: use is_fast assertion

    @JoanFM
    JoanFM committed May 6, 2024
    Configuration menu
    Copy the full SHA
    ef5dd36 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'master' of https://github.com/JoanFM/llama.cpp into fix… 

    …-pre-tokenizer-hashing
    @JoanFM
    JoanFM committed May 6, 2024
    Configuration menu
    Copy the full SHA
    3a4219a View commit details
    Browse the repository at this point in the history