QLoRA 4-bit quantization

[unbounded]

It would be interesting to compare this approach to the quantization in llama.cpp:

https://huggingface.co/blog/4bit-transformers-bitsandbytes

We present QLORA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU while preserving full 16-bit finetuning task performance. QLORA backpropagates gradients through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters (LoRA).

As I understand it, the main idea is to fine tune the model with a LoRA on each layer after 4-bit quantization, to restore performance to pre-quantization levels.

This could probably be applied to a GGML quantized model as well - either by doing the actual fine tuning in GGML or by training in Python and exporting the LoRA.

Some additional techniques they claim helps generation quality:

NormalFloat quantization

4-bit NormalFloat Quantization The NormalFloat (NF) data type builds on Quantile Quantization
[15] which is an information-theoretically optimal data type that ensures each quantization bin has an
equal number of values assigned from the input tensor.

This sounds similar to what @MarcioPais experimented with in #397 (comment), where they said:

It is however not worth it at all, as the non-linear mappings help with RMSE and MAE, but do basically nothing for improving perplexity, which is disappointing.

It is interesting that the paper calls this out as a clear improvement. Some possibilities I can think of:

• Quantile Quantization was not one of the non-linear mappings tried in Investigate alternative approach for Q4 quantization  #397
• Quantile Quantization does not directly help perplexity, but preserves information that can be used by the LoRA.

Double Quantization

Very similar to the super blocks @ikawrakow uses in #1256. The paper uses a 8-bit scale value for every 64 4-bit weights, and a 32-bit scale for every 256 8-bit scales.

Other notes

They don't show any results for 3-bit quantization, seems like an obvious next step.

[ggerganov]

This could probably be applied to a GGML quantized model as well - either by doing the actual fine tuning in GGML

I agree - this is a very interesting area for experiments.

User @xaedes has laid the foundation for training with the baby-llama example and is also making very interesting progress at full-text training: ggerganov/ggml#8 (comment)

CPU-based training / fine-tuning with quantization support could be very useful since it is much easier to afford a >128GB machine. We also have the mechanism to offload part of the computations to the GPU if necessary to get a bit of extra performance. In general, it looks like we have a good opportunity for demonstrating ggml-based fine tuning

It is interesting that the paper calls this out as a clear improvement.

The author actually acknowledged that GPTQ quantization is superior to NF4: https://twitter.com/Tim_Dettmers/status/1661482614811918338

[ikawrakow]

@unbounded

Double Quantization

Very similar to the super blocks @ikawrakow uses in #1256. The paper uses a 8-bit scale >value for every 64 4-bit weights, and a 32-bit scale for every 256 8-bit scales.

Yes, they did find the "super-block" or, as they call it, "double quantization" trick. But based on their numbers (model sizes after quantization), it does not look like they have found the best strategy. I haven't come around to be putting this stuff into llama.cpp (and the branch I'm working on is hopelessly out of sync with master, so I will have to eventually add it manually), but here here is a graph of perplexity vs model size that shows what is possible when one uses the bits wisely:

The graph shows perplexity as a function of quantized model size. At the end of the day, to me it looks like what matters most is how many bits were spent in total and how these bits were distributed between the various tensors. Minimizing some measure of difference between the original and quantized model weights does help, but it is a second order effect. I have added the current ggml Q4_0 and Q4_1 perplexities and the GPTQ numbers, taken from this post, to better illustrate the wide range in performance one gets for the same model size, depending on how the bits were spent. The "pure" 4-bit curve in black uses 4-bit quantization for all tensors except output.weight, which is quantized with a 6-bit variant. The different points depend on how the bits were distributed between the quants and the scales, and on how many blocks there are in a super-block. For the mixed 4-6 bit curve in red some of the tensors are quantized with 5 or even 6 bits. The x-axis is the quantized model size on disk in GiB (2^30 bytes).

You mention 3-bit quantization. Here is another graph where my current 3-bit results are included in orange.
These are "mixed" models, where most tensors are quantized with 3 bits, but some tensors use 4-6 bits.As we get below 3 GiB, perplexities rapidly grow with decreasing model size, so I have not included data in this range.

[unbounded]

You mention 3-bit quantization. Here is another graph where my current 3-bit results are included in orange. These are "mixed" models, where most tensors are quantized with 3 bits, but some tensors use 4-6 bits.As we get below 3 GiB, perplexities rapidly grow with decreasing model size, so I have not included data in this range.

Very cool!
There's some experiment's I'd like to run with a 3-bit quantization when I find the time, like storing a small amount of outliers per row/superblock. It's probably worth having another look at nonlinear quantizations as well.

[ikawrakow]

There's some experiment's I'd like to run with a 3-bit quantization when I find the time, like storing a small amount of outliers per row/superblock. It's probably worth having another look at nonlinear quantizations as well.

I had the same thought too, so I implemented outliers per super-block some time ago. Here is a graph comparing 3-bit quantization with and without outliers:

The black like is without using outliers. The point furthest to the left at about 2.75 GiB is for a "pure" 3-bit quantization, the others obtained by quantizing various subsets of the tensors with 4 or 5 bits. The red line is for 4 outliers, and the blue line for 8 outliers per super-block. The circled digits 1, 2, 3 indicate to which model the outliers have been added. As I did not see a way to implement a variable number of outliers per super-block that does not completely kill inference performance, in my implementation each super-block has a fixed number N of "outliers" (4 or 8 here), which are basically the N highest absolute value weights. They are replaced with zeros in the 3-bit encoding, and are added to the super-block using 8-bit quantization. We need 16 bits per outlier (position in the super-block + quantized value), plus another 16 bits for the outlier scale. This works out to 0.3125 extra bits per weight for 4 outliers, and 0.5 bits per weight for 8 for a super-block of 256 weights.

As we can see from the graph, separating outliers does improve accuracy, but it does so at a less efficient rate compared to using more bits for some of the tensors. On the other hand this implementation is far from perfect. My observation is that "outliers" tend to cluster in some portions of the tensors, so most of the time we are basically wasting bits in super-blocks that don't have real "outliers", while not being able to encode all real outliers in some super-blocks having more than N real outliers. So, I think, it would be better to have outliers per row or even per tensor. But as far as I can tell, this would require a much bigger change to ggml compared to the simple per super-block implementation. I was discussing this issue just yesterday with @ggerganov, and his response was that the easiest way to add per tensor outliers is to add a new tensor type along with a new ggml operation that performs the sparse outlier matrix multiplication. My gut feeling is that this would be fine for single token prediction, but will reduce performance in batch mode because intermediate results need to be fetched from memory twice, instead of once, and in batch mode they for sure don't fit in the L1 cache (or maybe even L2 cache, depending on model and batch size).

[phdykd]

I am getting this issue " No GPU found. A GPU is needed for quantization." for the following code snippet trying to be implemented in M2 MacOS that has 12 CPU and 38 GPU in it. How is QLORA/quantization will work on M2 MacOS systems that has "mps" in it?

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig

model_id = "EleutherAI/gpt-neox-20b"
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_use_double_quant=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16
)

tokenizer = AutoTokenizer.from_pretrained(model_id)

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config) #, device_map={"":0})

model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config, device=torch.device('cpu'))