Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Decide pre tokenizer based on preprocessing of entry and not on tokens encoded #7039

Closed
wants to merge 6 commits into from
Closed

Decide pre tokenizer based on preprocessing of entry and not on tokens encoded #7039

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
a1aa65e
feat: change convert hf to gguf
JoanFM May 2, 2024
0672cd8
use conver ids to tokens
JoanFM May 2, 2024
0f94ff7
fix: only do pre tokenization and normalization
JoanFM May 2, 2024
c7a6c32
fix: protect against slow tokenizer
JoanFM May 2, 2024
ef5dd36
fix: use is_fast assertion
JoanFM May 6, 2024
3a4219a
Merge branch 'master' of https://github.com/JoanFM/llama.cpp into fix…
JoanFM May 6, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
8 changes: 6 additions & 2 deletions convert-hf-to-gguf.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -273,8 +273,12 @@ def get_vocab_base_pre(self, tokenizer) -> str:

chktxt = '\n \n\n \n\n\n \t \t\t \t\n \n \n \n \n🚀 (normal) 😶\u200d🌫️ (multiple emojis concatenated) ✅ 🦙🦙 3 33 333 3333 33333 333333 3333333 33333333 3.3 3..3 3...3 កាន់តែពិសេសអាច😁 ?我想在apple工作1314151天～ ------======= нещо на Български \'\'\'\'\'\'```````""""......!!!!!!?????? I\'ve been \'told he\'s there, \'RE you sure? \'M not sure I\'ll make it, \'D you like some tea? We\'Ve a\'lL'

chktok = tokenizer.encode(chktxt)
chkhsh = sha256(str(chktok).encode()).hexdigest()
if tokenizer.is_fast:
chktok = tokenizer.backend_tokenizer.pre_tokenizer.pre_tokenize_str(tokenizer.backend_tokenizer.normalizer.normalize_str(chktxt))
chkhsh = sha256(str(chktok).encode()).hexdigest()
else:
chktok = tokenizer.encode(chktxt)
chkhsh = sha256(str(chktok).encode()).hexdigest()

logger.debug(f"chktok: {chktok}")
logger.debug(f"chkhsh: {chkhsh}")
Expand Down