PEFT and LoRa for MLX?

[power9799]

How do I run code for PEFT and LoRa in MLX? I want to fine a mistral model locally.

[awni]

Here's some docs on using LoRA with MLX LM.

[power9799]

Thanks!

[power9799]

Suppose I can get the fine tuning to work, how do I export a model fine tuned on MLX to run on Pytorch with CUDA ?

[awni]

After fine-tuning, fuse the adaptors into the base model which will save it in safetensors. From there you should be able to load it and use it with Hugging Face Transformers (which uses PyTorch).

For a more detailed guide check out e.g. https://apeatling.com/articles/part-4-testing-and-interacting-with-your-fine-tuned-llm/

[power9799]

I can't find the adapter.npz under the mlx-examples folder, I used mdfind "adapter.npz" still couldn't find it. Is it the LoRa file? I also I used your code, it gave this error, Error while finding module specification for 'mlx_lm.lora' (ModuleNotFoundError: No module named 'mlx_lm') Can't someone update MPS for pytorch more often?

[awni]

Error while finding module specification for 'mlx_lm.lora'

python -m pip install -U mlx-lm

I can't find the adapter.npz under the mlx-examples folder, I used mdfind "adapter.npz" still couldn't find it.

They will be saved wherever you run the command from