1bit+ quantizing idea #5164
Swight1423
started this conversation in
Show and tell
Replies: 1 comment
-
looking like my accuracy was largely due to hovering near the average. just taking the average between the max and min values for everything would yield a similar result. may continue working on the problem but future solutions will likely take a little bit more data to constrain to better accuracy levels. |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
1bitquant.zip
I took a bit different approach to the problem of shrinking down models. instead of doing a direct calculation I instead store if each value is less than the next and guess based on that information for a series of values. the quant method assumes integer values like you would get if the model was already quantized may still need the group delta values from the original model. I am pretty sure there is still room for improvement over my current implementation linked to this post. my implementation in vb.net just generates randomized data in a given range, finds the comparison data, attempts to use that data to reconstruct for the selected group size, then compares to the original and averages multiple runs. the part of the code that actually does the work is less than 100 lines of code. There is a GUI to mess with the values. if my calculations are accurate could get 30%ish accuracy when run on 4bit data and 24%ish when run on 8bit or 16bit as currently coded on average. haven't tested on real models. suggest group sizes greater than 32. needs more testing to find sweet spot.
Edit: 8bit sweet spot seems to be around 32768 for group size where I got around 31% accuracy.
Edit2 looks like my accuracy calculation may have issues but they seem to translate to this being better than the implementations I removed. the calculation was more a within what percentage of the correct answer do I get if I didn't make another mistake.
Beta Was this translation helpful? Give feedback.
All reactions