Replies: 22 comments 24 replies
-
|
I'm surprised that importance matrices trained on 10k/100k/1m token barely seem to diverge from each other. While some “overfitting” does seem to occur, it's also significantly less prevalent than one might expect. |
Beta Was this translation helpful? Give feedback.
-
|
I think the median / average might not be the smartest measurement. Here's KLD_99 sorted for your q2_K files: Btw, thank you for helping investigate this, I've been very curious about optimal quantization calibration. |
Beta Was this translation helpful? Give feedback.
-
|
Working on a script right now that will automatically quantize a bunch of randomized groups of text data and measure KLD_99. |
Beta Was this translation helpful? Give feedback.
-
|
Please share the script, if you can! |
Beta Was this translation helpful? Give feedback.
-
|
@Artefact2 I have a question. |
Beta Was this translation helpful? Give feedback.
-
|
group_40.txt If I use the first 25k tokens of this data: If I use 40k tokens of Wikitext: I am measuring KL divergence over 30,000 tokens that uses a mix of data (Lyrics, conversations, a wikipedia article or two, etc), so the sample size should be large enough to rule out any differences there. What especially improves are the harder to predict outliers, as noted by KLD_99 and KLD_95. Both are q2_K quantized using the base model of Fett-uccine-7B-GGUF. What doesn't improve over Wikitext is PURELY random data (it's actually a little bit worse); pseudo-random data seems to be optimal, though. |
Beta Was this translation helpful? Give feedback.
-
What's the argument for using an unrelated model instead of the F16 model that you are quantizing? |
Beta Was this translation helpful? Give feedback.
-
No particular one; as long as the data is sufficiently high entropy while still vaguely coherent, this technique should be roughly similar in performance. It might be even better to generate from the source model. |
Beta Was this translation helpful? Give feedback.
-
|
This is about ~50k pseudo-random tokens. I recommend using this file for doing imatrix calibration from here on out; imatrix data should generally transfer well across different models. |
Beta Was this translation helpful? Give feedback.
-
|
@Artefact2 Nice work! On using (pseudo-) random data for imatrix generationIf you have a meaningful calibration dataset, I recommend against (pseudo-)random data. A more comprehensive evaluation that does not rely on a single, quite small dataset will tend to favor imatrix created from textual data. Here is an example:
I'm not using Winogrande and ARC because the test/validation datasets that I have available for these are too small to reveal statistically significant differences. The table summarizes the results
My take from this data
|
Beta Was this translation helpful? Give feedback.
-
|
Out of curiosity, are you running imatrix with the default settings besides |
Beta Was this translation helpful? Give feedback.
-
One thing that I did note is that pretraining style data (non-wikitext) outperformed Wikitext (especially for maximum KL div error reduction), but didn't quite reach the pseudo-random data's KL divergence statistics. Also, if attention tensors are sensitive to calibration data, it might be optimal to use full context for calibration instead of 512 ctx. The main reason why I'm suspicious on wikitext is that it doesn't contain any code or programming examples and I think that skews it towards natural english. |
Beta Was this translation helpful? Give feedback.
-
|
I am using 200k worth of tokens randomly sampled from MiniPile, a decent small pretraining dataset, and I'm comparing different context sizes for calibration (q4_K_S, 7b): 2048 ctx calibration, 200k tokens ish pretraining data 4096 ctx calibration, 200k tokens ish pretraining data 8192 ctx calibration, 200k tokens ish pretraining data 512 ctx calibration, 200k tokens ish pretraining data It would seem that native context size is the best all around for evaluating the importance matrix and results in the lowest average divergence, as well as the best outlier error reduction. |
Beta Was this translation helpful? Give feedback.
-
|
groups_merged.txt Next up, I will be looking into a way to preselect individual pieces of a larger dataset for higher KL div / higher PPL outlier sections so that the quantization is more robust to outliers (instead of throwing random data at it). |
Beta Was this translation helpful? Give feedback.
-
|
Nice! Yeah, that's definitely a good improvement, thank you again for these tests. It seems like V2-TurboMini is doing a little bit worse compared to V2. I don't know if that's due to the dataset or the chunk size. But yeah for the best user experience, I would recommend using all of that imatrix data with the max chunk size. So I would recommend one of the three datasets that gets the best result at max chunk size. Would be great if you could test the others at max chunk size as well, and lastly compare it to groups_merged at max chunk size. That way we can determine which one the best is. Are you using Windows? I do have issues with llama.cpp exiting as well sometimes. Specifically when I try to run my copy pasted test prompts. Hopefully you will be able to sort it out, but even if not, I think those results are applicable to L3 as well. I've also run some perplexity tests and the improvements were similar like shown with 7B. |
Beta Was this translation helpful? Give feedback.
-
|
Hey - finished re-running this for enhancedv2 with max chunks and updated the screenshot above with the latest. And to complete the set here are the IQ2_XXS results for groups_merged-enhancedV2-MegaTurboDerp.txt: As for Groups_merged.txt, the results did not change when setting chunks to max. This can be observed above if you compare the 105 chunks groups_merged.txt IQ2_XXS run with the max chunks IQ2_XXS run right below it, so for simplicity's sake I'll omit it here as the results are identical again. It makes sense because groups_merged.txt has 105 chunks, so setting it to max would not make a difference versus capping at 105. And yeah tried entering in the commands manually as well to make sure but still no dice. Going to go through the steps for L3-8b again later today but this should give some insight to how the three files perform versus groups merged at least. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you again! Based on your results, I would recommend MiniTurbo. It improves nicely over Groups Merged while not taking quite as long to compute as the Mega one. Would be really cool if you could get L3 running, maybe this time you could test a higher quant like q4_k_s, as that one is quite popular. But even if you don't get it to work, your results were already very insightful, so thank you for that! |
Beta Was this translation helpful? Give feedback.
-
|
The MiniTurbo seems like the best of the three from what we've seen above. While the compute for Mega isn't too bad for something like a 7b it will naturally ramp up significantly with larger sizes. In terms of L3, opened an issue here for it. Figured this would be better for visibility in case others are experiencing the same. Will try running with q4_k_s once we make headway there. And no problem at all, happy to help. There's always more research to do when it comes to how different kinds of data affect the importance matrix calculations but it's refreshing to see gains in a process that has more or less become standardized over the past few months. Thank you again for sharing this. |
Beta Was this translation helpful? Give feedback.
-
|
Found a workaround for converting the arc-challenge-validation binary file to .json so perplexity.exe can still execute as intended. Here's all 4 results for Llama-3-8b @ Q4_K_S with max chunks. At a quick glance it looks like enhancedv2 showed a better Max KLD than groups_merged.txt and about the same KLD mean if you account for the error margin. Interested to hear your thoughts as well. Groups_merged.txt (Control) EnhancedV2 Enhancedv2-TurboMini enhancedV2-MegaTurboDerp.txt |
Beta Was this translation helpful? Give feedback.
-
There are indeed chunks that have higher outlier frequencies compared to the rest of the data. This is (500k?) tokens ish of wiki.train.raw. I used "KLD_95" (so, the average KL div for the top 5% highest KL divergences). (I clustered them for chart readability.) As you can see, there are a lot of chunks on the "tail end" that are less affected by quantization: the top 5% highest KL div was 0.937 for the very worst chunk, and 0.214 for the best chunk. My theory is that selecting the top % of chunks that are most "damaged" by quantization for calibration reduces outlier error better than pseudo-random data would. And this way, we have a more mathematically "precise" way of finding better calibration data compared to just throwing 100k tokens of either unsorted Wikitext or gibberish at it. |
Beta Was this translation helpful? Give feedback.
-
|
Pre-selecting in this manner does indeed improve the quantization end result. No imatrix q4_0: ===== KL-divergence statistics Top ~40k tokens, sorted by top5% highest avg kl divergence imatrix, q4_0 ===== KL-divergence statistics All ~500k tokens wikitext data, imatrix, q4_0: ===== KL-divergence statistics |
Beta Was this translation helpful? Give feedback.
-
|
@ikawrakow , @Artefact2 , considering the benefits of the iMatrix, not only in English, but also in other languages like my very own French, could one of you guys assemble and share a training file of alternated English / French (and why not with most of the languages broadly supported to some extend by the Llama2 & Mistral models) sequences of text allowing to train properly an iMatrix benefiting all the languages involved? And an eval file of each language in the fashion of wiki.test.raw? Ideally, it would work like wiki.train.raw does no matter the ctx chosen (I use 32 and it works quite well, but 128 or 512 are still probably a bit better) and the numbers of chunks chosen, up to a few thousands by language. For example, if I set 500 chunks on the iMatrix, 250 chunks would be trained in English, 250 chunks in French. I lack of the know how to do that properly and efficiently involving all the aforementioned criteria. But I think it would help greatly a lot of people to make a single iMatrix file and a single series of quant benefiting a maximum amount of people, including yours truly. |
Beta Was this translation helpful? Give feedback.
-
|
is there data that shows the difference between the 20k random dataset and the new pseudo-random data? |
Beta Was this translation helpful? Give feedback.
-
|
Yes, there is data that supports the current assumption that the pure random dataset leads to worse results than the new balanced random data. However, there is also an indication that we are looking at noisy results and that the new balanced random data is also not optimal. Here is one of the Reddit threads with test results. |
Beta Was this translation helpful? Give feedback.
-
|
Also wondering! From
Also wondering this. At least in theory it could have a big impact what is used. I guess the only way to really test this idea is to create the same model with different data sources for the imatrix and compare ppl? Or is there another way to do this? |
Beta Was this translation helpful? Give feedback.
-
|
In addition to the link in the reply above there are more test results with perplexity and hellaswag in this Reddit thread. Be sure to expand the whole thing, as Reddit likes collapsing replies. |
Beta Was this translation helpful? Give feedback.
-
|
@ikawrakow Thanks for all the hard work, great job. The link to wiki.train.raw is down at https://s3.amazonaws.com/research.metamind.io/wikitext/wikitext-2-raw-v1.zip?ref=blog.salesforceairesearch.com do you know an alternate link, thanks. |
Beta Was this translation helpful? Give feedback.
-
|
I have added them to this repository: https://huggingface.co/datasets/ikawrakow/validation-datasets-for-llama.cpp |
Beta Was this translation helpful? Give feedback.
-
|
Thanks again, what context length would you recommend for creating the imatrix.dat file using wiki-train.raw for 70b models and mixtral? |
Beta Was this translation helpful? Give feedback.
-
|
I use 512 tokens most of the time. There has been a lot of discussion around the imatrix topic, but for me the small improvements that one may get from using different context lengths, different calibration datasets, etc., are very minor compared to what one gains from using vs not using an imatrix. |
Beta Was this translation helpful? Give feedback.
-
|
Has anyone experimented with adding a small value to the importance matrix weights? If I have understood correctly, the importance matrix weights are an approximate diagonal Hessian. If that is the case, a common way to deal with overfitting is to add a scalar multiple of the identity matrix to the (diagonal or otherwise) Hessian (see: Tikhonov_regularization). The regularized Hessian matrix where To determine the optimal value of
From a Bayesian perspective:
It's likely the use of random and semi-random data mentioned in this thread is acting as a "quick and dirty" form of regularisation anyway: https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/bishop-tikhonov-nc-95.pdf and IMO, it would probably be better to consider doing it in a more principled way - especially considering the calibration dataset is so small and the imatrix computation isn't using the full context nor the correct prompt format, etc. Also, has anyone actually looked into a full Hessian approximation or at least checked the off-diagonals are small? If there is any significant multicollinearity among the weights then this method of using only the diagonal Hessian like this could have a serious impact on the model. How feasible is it to compute the full Hessian approximation by summing |
Beta Was this translation helpful? Give feedback.
-
lol |
Beta Was this translation helpful? Give feedback.
-
People put a lot of weight on it being English (and not code or another language) but I think the biggest problem is the data leakage between If you don't know what I mean by "data leakage" then this might help: https://gwern.net/tank There are ways we could correct for this bias in the reported drops in PPL, but most are nonparametric and would require quite a lot of extra computation... |
Beta Was this translation helpful? Give feedback.
-
I was just laughing at wikipedia being "factual." I know what was meant. It was a half joke. On a serious note though, we probably shouldn't be using wikipedia (I'm currently using it too). I don't know how much the gross political bias matters for imatrices, but these LLM's being trained on things like wikipedia and reddit concerns me |
Beta Was this translation helpful? Give feedback.
-
|
Just trying to wrap my head around how these weights are used: So So if and the unweighted version is just: So I'm trying to see how What is special about We can also look at |
Beta Was this translation helpful? Give feedback.
-
|
Edit: Never mind. Not using the latest llama.cpp with the latest quantizations of Llama 3 8b Instruct and with the right settings were the cause the hallucination spike I was seeing. |
Beta Was this translation helpful? Give feedback.
-
Nope, I've looked at this all sorts of ways and can't for the life of me see where the use of static void quantize_row_q4_0_impl(const float * restrict x, block_q4_0 * restrict y, int64_t n_per_row, const float * quant_weights) {
static_assert(QK4_0 == 32, "QK4_0 must be 32");
if (!quant_weights) {
quantize_row_q4_0_reference(x, y, n_per_row);
return;
}
float weight[QK4_0];
int8_t L[QK4_0];
float sum_x2 = 0;
for (int j = 0; j < n_per_row; ++j) sum_x2 += x[j]*x[j];
float sigma2 = sum_x2/n_per_row;
const int64_t nb = n_per_row/QK4_0;
for (int ib = 0; ib < nb; ++ib) {
const float * xb = x + QK4_0 * ib;
const float * qw = quant_weights + QK4_0 * ib;
for (int j = 0; j < QK4_0; ++j) weight[j] = qw[j] * sqrtf(sigma2 + xb[j]*xb[j]);
float d = make_qx_quants(QK4_0, 8, xb, L, 1, weight);
y[ib].d = GGML_FP32_TO_FP16(d);
for (int j = 0; j < 16; ++j) {
y[ib].qs[j] = L[j] | (L[j+16] << 4);
}
}
}For this to be consistent:
All I can think is that the Reformulating this to use a proper Also the code in Even just looking for the //float sum_x2 = 0;
//for (int j = 0; j < QK_K; ++j) sum_x2 += x[j]*x[j];
//float sigma2 = sum_x2/QK_K;
float max_scale = 0;
float max_abs_scale = 0;
for (int ib = 0; ib < QK_K/16; ++ib) {
float scale;
if (quant_weights) {
const float * qw = quant_weights + QK_K*i + 16*ib;
//for (int j = 0; j < 16; ++j) weights[j] = qw[j] * sqrtf(sigma2 + x[16*ib + j]*x[16*ib + j]);
//scale = make_qx_quants(16, 32, x + 16*ib, L + 16*ib, 1, weights);
scale = make_qx_quants(16, 32, x + 16*ib, L + 16*ib, 1, qw);
} else {
scale = make_qx_quants(16, 32, x + 16*ib, L + 16*ib, 1, NULL);
}
scales[ib] = scale;and so on... It's not as though this code has to be really efficient either. All the sections of code like the above could easily be refactored into a single function instead of repeating it with slight tweaks 15-20 times. |
Beta Was this translation helpful? Give feedback.
-
|
The The Unreasonable Ineffectiveness of the Deeper Layers paper as implemented by Charles Goddard in PruneMe, suggest that a lot of the later blocks are doing very little and a refactored version of |
Beta Was this translation helpful? Give feedback.
-
Imatrix has been here for a while and I haven't seen many guidelines (or testing at all) on how to use it. Common objections/concerns are overfitting, and generating the imatrix on the "wrong" kind of text. See #5006
To try and gather some data, I tried three datasets for training/testing and three different number of chunks (10K = 20 chunks of 512 tokens, 100K = 200 chunks, 1M = 2000 chunks) used for calculating the imatrix.
frwikiis part of a raw XML dump of the french Wikipedia. It contains a mix of structured XML data, french text, and wikicode markup.mbotfis concatenated text of my Malazan Book of the Fallen books. It's english fiction.wikiis the wikitext we all know and seem to use. It contains factual english text.I used Mistral-7B to calculate KL-divergence median for Q2_K quants generated with all nine possible imatrixes on all three test datasets, and no imatrix as a baseline.
Looking forward to your opinions on the results, or about the methodology. For now, I'll keep using wikitext with 100K tokens. Might not always be optimal depending on the model's use case, but it seems unlikely to make things worse.
Raw data: imatrix-tests.zip
Beta Was this translation helpful? Give feedback.
All reactions