Use String, Dict, and read_bytes to shorten and simplify #91
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is based off current nightly branch (mojo 2024.4.161). It is a demo of some clean ups that can happen now that Mojo and its stdlib have added a lot of functionality that was missing when this was originally released. There could probably also be another round to remove TensorSlice and just use
List[TensorF32]
for each layer of weights.The main changes are:
read_bytes
to handling of pointers without copying. That means FileBuf is no longer used.I am not sure this is ready to merge mostly because the of handling of special bytes handling in
wrap
and the old print function. I tried to persevere the functionality but haven't tested extensively. Ideally we could get proper handling from String and if not fix it in stdlib.Also, I think the stdlib is going to shift to
List[UInt8]
for all bytes representations, including in String. So this change could also wait until after has happened and is incorporated.I didn't mess with Llamatune since this is going across Mojo versions but locally there was no change in tokens / sec. It is probably loading faster and more memory efficiently since this avoids the vocab sort and no longer reads entire
tokenizer.bin
.