Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

tests : add test-tokenizer-0.sh #7036

Merged
merged 15 commits into from
May 4, 2024
Merged

tests : add test-tokenizer-0.sh #7036

merged 15 commits into from
May 4, 2024

Commits on May 2, 2024

  1. tests : add test-tokenizer-0.sh

    @ggerganov
    ggerganov committed May 2, 2024
    Configuration menu
    Copy the full SHA
    ce7d3a0 View commit details
    Browse the repository at this point in the history

  2. unicode : add all unicode number ranges

    @ggerganov
    ggerganov committed May 2, 2024
    Configuration menu
    Copy the full SHA
    7053b26 View commit details
    Browse the repository at this point in the history

  3. starcoder : fix pre-tokenizer

    @ggerganov
    ggerganov committed May 2, 2024
    Configuration menu
    Copy the full SHA
    cf00fe1 View commit details
    Browse the repository at this point in the history

  4. tests : add test that fails with DeepSeek tokenizers

    @ggerganov
    ggerganov committed May 2, 2024
    Configuration menu
    Copy the full SHA
    3a461db View commit details
    Browse the repository at this point in the history

  5. falcon : fix regex

    @ggerganov
    ggerganov committed May 2, 2024
    Configuration menu
    Copy the full SHA
    3275e60 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2024

  1. unicode : regenerate unicode tables

    @ggerganov
    ggerganov committed May 3, 2024
    Configuration menu
    Copy the full SHA
    cd7c728 View commit details
    Browse the repository at this point in the history

  2. refact : add tokenizer model

    @ggerganov
    ggerganov committed May 3, 2024
    Configuration menu
    Copy the full SHA
    d53240c View commit details
    Browse the repository at this point in the history

  3. lint : fix

    @ggerganov
    ggerganov committed May 3, 2024
    Configuration menu
    Copy the full SHA
    c30056a View commit details
    Browse the repository at this point in the history

  4. tests : disable failing tests 

    ggml-ci
    @ggerganov
    ggerganov committed May 3, 2024
    Configuration menu
    Copy the full SHA
    bc26eb7 View commit details
    Browse the repository at this point in the history

  5. refact : add tests files 

    ggml-ci
    @ggerganov
    ggerganov committed May 3, 2024
    Configuration menu
    Copy the full SHA
    9745cf8 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2024

  1. Merge branch 'master' into gg/add-tokenizer-test-script

    @ggerganov
    ggerganov committed May 4, 2024
    Configuration menu
    Copy the full SHA
    26f606e View commit details
    Browse the repository at this point in the history

  2. convert : print -> logging 

    ggml-ci
    @ggerganov
    ggerganov committed May 4, 2024
    Configuration menu
    Copy the full SHA
    d974aed View commit details
    Browse the repository at this point in the history

  3. lint : fix

    @ggerganov
    ggerganov committed May 4, 2024
    Configuration menu
    Copy the full SHA
    5f30e30 View commit details
    Browse the repository at this point in the history

  4. unicode : digit -> number

    @ggerganov
    ggerganov committed May 4, 2024
    Configuration menu
    Copy the full SHA
    f19b45c View commit details
    Browse the repository at this point in the history

  5. phi-3 : update

    @ggerganov
    ggerganov committed May 4, 2024
    Configuration menu
    Copy the full SHA
    7e11d40 View commit details
    Browse the repository at this point in the history