CodeLlama SFT

[weirukai]

We fine-tuned CodeLlama-7b-instruct with the provided SFT code. The configuration is as follows:

--model_name_or_path
./CodeLlama-7b-Instruct-hf
--do_train
--dataset
example_text2sql_train
--max_source_length
2048
--max_target_length
512
--finetuning_type
lora
--lora_target
q_proj,v_proj
--template
llama2
--lora_rank
64
--lora_alpha
32
--output_dir
./dbgpt_hub/output/adapter/CodeLlama-7b-Instruct-sft-lora
--overwrite_cache
--overwrite_output_dir
--per_device_train_batch_size
1
--gradient_accumulation_steps
16
--lr_scheduler_type
cosine_with_restarts
--logging_steps
50
--save_steps
1000
--learning_rate
2e-4
--quantization_bit
4
--num_train_epochs
8
--plot_loss
--bf16
--dataloader_num_workers
12

We got very performance since the model answers the question with some sentence not related to SQL command, e.g.

---

To find the number of employees hired in each shop, you can use the following SQL command:

SELECT s.Name, COUNT(e.Employee_ID) AS Num_Employees
FROM shop s
JOIN hiring h ON s.Shop_ID = h.Shop_ID
JOIN employee e ON h.Employee_ID = e.Employee_ID
GROUP BY s.Name;

This command joins the shop table with the hiring table on the Shop_ID column, and then joins the resulting table with the employee table on the Employee_ID column. The COUNT function is used to count the number of employees hired in each shop, and the resulting table is grouped by the Name column of the shop table.

The resulting table will have two columns: Name and Num_Employees. The Name column will contain the name of each shop, and the Num_Employees column will contain the number of employees hired in each shop.

---

Could you help me to figure out where is the problem? Thanks!!!!