You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I know the tokenizer.py in this Repo use TikTokenizer, can I use transformers.AutoTokenizer to load the tokenizer so that I dont need to amend my code class? And if i not use tokenizer.py, ChatFormat can not be used too.
The text was updated successfully, but these errors were encountered:
I know the tokenizer.py in this Repo use TikTokenizer, can I use transformers.AutoTokenizer to load the tokenizer so that I dont need to amend my code class? And if i not use tokenizer.py, ChatFormat can not be used too.
The text was updated successfully, but these errors were encountered: