Load in 4-bit only not enough for Cog with 12 Gb vram

[karachay-b]

Hi Peeps,
before the monkey patch was integrated in the loading procedure of cog-models, I was able to run those on my 12 Gb geforce with the following additional bitsandbytes-arguments:

• torch.dtype = torch.bfloat16
• bnb_4bit_quant_type="nf4"
• bnb_4bit_use_double_quant=True

Now I'm always OOM whether 4bit is checked or not. (using one beam)

Does anyone have tips for me how to optimize the loading to be again able to run cog-models?

@jhc13
a tip where to put the additional arguments in the new loading process would also be appreciated. I've already tried my luck, but with no avail.

Thanks

[jhc13]

before the monkey patch was integrated in the loading procedure of cog-models,

What change are you talking about exactly? The Cog models always required some monkey-patching.

I was able to run those on my 12 Gb geforce with the following additional bitsandbytes-arguments:

torch.dtype = torch.bfloat16
bnb_4bit_quant_type="nf4"
bnb_4bit_use_double_quant=True

How did you provide these arguments? Did you edit TagGUI's source code?

[karachay-b]

I meant version 1.12.0. before monkey_patch() was an external function in auto_captioner.py and the function load_processor_and_model() was defined in the same file.

In version 1.12.0 in auto_captioner.py at line 458:

if load_in_4_bit:
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_quant_type="nf4",
bnb_4bit_use_double_quant=True)
dtype_argument = {}

[jhc13]

Try editing line 160 of captioning_thread.py.

[karachay-b]

Thank you for the fast replies. Ive changed the code in said location.
But sadly only THUDM/cogagent-vqa-hf is working. For the oder Cog-model im 20 mb short.
But it'll do.