mlx 0.13 very slow with q8 and fp16

[ivanfioravanti]

I was testing the new quantize by @angeloskath with some Italian prompts that were failing with previous version and now are PERFECT! But while doing this I have seen extreme slowness with q8 and fp16 version.

I'm using meta-llama/Meta-Llama-3-70B-Instruct for test on M2 Ultra 192GB.

I created 3 conversions of it with:

mlx_lm.convert --hf-path meta-llama/Meta-Llama-3-70B-Instruct -q --q-bits 4 --mlx-path mlx_model_q4
mlx_lm.convert --hf-path meta-llama/Meta-Llama-3-70B-Instruct -q --q-bits 8 --mlx-path mlx_model_q8
mlx_lm.convert --hf-path meta-llama/Meta-Llama-3-70B-Instruct --mlx-path mlx_model_fp16

Then I tested generation with:

mlx_lm.generate --prompt "Write the story of Albert Einstein" --model mlx_model_q4
mlx_lm.generate --prompt "Write the story of Albert Einstein" --model mlx_model_q8
mlx_lm.generate --prompt "Write the story of Albert Einstein" --model mlx_model_fp16

q4 14.9 t/s (GPU >1300Mhz 115W)
q8 & fp16 0.4 t/s (GPU <1100Mhz peak 2W)

[ivanfioravanti]

I tested on M3 Max and got same result. with even lower power consumption and GPU frequency. Really strange.

[awni]

It's pretty slow for me also in 8-bit (and presumably 16 as well, but didn't test). Not sure why yet.

[awni]

@ivanfioravanti just curious you phrased this issue as if it used to be faster in previous MLX versions. Is that the case?

[ivanfioravanti]

Sorry for the delay I was out the whole week and happy to be back playing with MLX 🛝
I thought the issue was impacting all models using q8 but this is not the case. It seems issue is present with Llama 70B only.

• Mistral-7B-Instruct-v0.2 works like a charm
• mlx-community/dolphin-2.9.1-yi-1.5-34b-8bit works like a charm

Can this be related to model size? 70B? I will try comparing another large model.

[awni]

Can this be related to model size? 70B?

Oh yes, it's almost certainly related to the size / amount of RAM required. There seems to be a performance cliff for very large models. It shouldn't be swapping because its still not using all the RAM on the machine but it does seem related to memory page demand. Still debugging.. this one might take a little while to iron out.