v0.5.0: Agent Tuning, Unsloth Integration

Congratulations on 10k stars 🎉 Make LLM fine-tuning easier and faster together with LLaMA-Factory ✨

New features

• Support agent tuning for most models, you can fine-tune any LLMs with --dataset glaive_toolcall for tool using #2226
• Support function calling in both API and Web mode with fine-tuned models, same as the OpenAI's format
• LLaMA Factory 🤝 Unsloth, enjoy 170% LoRA training speed with --use_unsloth, see benchmarking here
• Supports fine-tuning models on MPS device #2090

New models

• Base models
  • Phi-2 (2.7B)
  • InternLM2 (7B/20B)
  • SOLAR-10.7B
  • DeepseekMoE-16B-Base
  • XVERSE-65B-2
• Instruct/Chat models
  • InternLM2-Chat (7B/20B)
  • SOLAR-10.7B-Instruct
  • DeepseekMoE-16B-Chat
  • Yuan (2B/51B/102B)

New datasets

• Supervised fine-tuning datasets
  • deepctrl dataset
  • Glaive function calling dataset v2

Core updates

• Refactor data engine: clearer dataset alignment, easier templating and tool formatting
• Refactor saving logic for models with value head #1789
• Use ruff code formatter for stylish code

Bug fix

• Bump transformers version to 4.36.2 by @ShaneTian in #1932
• Fix requirements by @dasdristanta13 in #2117
• Add Machine-Mindset project by @JessyTsu1 in #2163
• Fix typo in readme file by @junuMoon in #2194
• Support resize token embeddings with ZeRO3 by @liu-zichen in #2201
• Fix #1073 #1462 #1617 #1735 #1742 #1789 #1821 #1875 #1895 #1900 #1908 #1907 #1909 #1923 #2014 #2067 #2081 #2090 #2098 #2125 #2127 #2147 #2161 #2164 #2183 #2195 #2249 #2260