-
Notifications
You must be signed in to change notification settings - Fork 715
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
LFX Mentorship (Jun-Aug, 2024): Finetune LLM models for Rust coding assistance #3371
Comments
Hi @juntao! Could you please let me know if there's a pretest or any other steps I should take to participate? |
@juntao |
@juntao Is there any community(Discord, slack..) where we contributors can interact? What are the timelines and selection criteria for this project I am really excited about the project? |
Hey @juntao | @hydai I will go through all the resources , links mentioned above and update you both with my progress here to improve my chances of getting selected for the LFX program. |
ProgressI have started working and exploring the first project objective that is:
What I plan to do:
What all I did:I was able to complete all the planned steps I have mentioned above, below are my findings till now about the current PR review Bot. Findings | please verify this part (@juntao @hydai )
Issues
|
NextI'm currently exploring the Fine-tuning part, I will try to setup the fine-tuned chemistry assistant here which you provided and make some changes and understand its working. Also, I will be exploring the llama.cpp model, its working , docs, to get better understand and update you all with any further progress.
|
Hello @juntao @hydai , myself Akshat Shrivastava sophomore at IIT BHU(Varanasi) in past few days i have understood the workflow of finetuning and running llm's locally, i was wondering weather for current task i could use llama-3-8b-bnb-4bit as its a direct modification of llama3 8b but in 4 bit to run everything in my current hardware smoothly |
Hey @codemaster1104 , For the tool, llama.cpp is preferred one, because of its capability to run on CPU itself, also the resources required for fine-tuning will be provide by Organization itself. I hope I cleared some of your doubts !! |
Hi, @juntao. I am interested in this project. I went through the example for fine-tuning the model and also used the LlamaEdge API server to run the chat in my browser. I did not face any major issues while going through this, although I was initially confused about how LlamaEdge API works, this demo video - https://www.youtube.com/watch?v=KTquzmXVj9o from KubeCon helped me see concretely what we expect when running this. Do you suggest anything else I should look into to understand the tasks better? |
Progress:
llama-3-8b-Instruct-GGUF -test setup |
@juntao could you please guide us with the metrics and tools we could use to compare models , currently i am assuming that we would be manually comparing a bunch of answers for specific questions in both models, but i am sure there would be a better way |
@juntao @hydai really need your guidance here is such direct behaviour desired?, according to me for pr review bot should give on point explanations, If this behaviour is desired i will start working on my final dataset with similar types of explanations to code. |
|
Hey @juntao I have keen interest in this project. I have good hands-on experience with Large language models and made several project similar to this QNA project, which utilizes the OpenAI API and answers the latest answers. I would really be glad to in this project and refining my knowledge in this project. Could you tell me how I could interact with the members and other people.? |
My model for objective 2 is still hallucinating and it seems to get stuck on the context of previous questions, I would like to fine tune this model with recommended parameters first for which i will need more computing resources, I have also started to read papers on the topic to get better understanding. My progress
|
Is there any test planned for this project? |
Summary
WasmEdge is a lightweight inference runtime for AI and LLM applications. We want to build specialized and finetuned models for WasmEdge community. The model should be supported by WasmEdge and its applications should benefit the WasmEdge community.
In this project, we will build and compare two finetuned model for Rust coding assistance.
Details
Objective 1: Code review model
Create a dataset with the following two fields
We are looking for at least 200 Q and A pairs. The total length of each QA should be less than 3000 words.
The QA could come from Rust documentation such as Rust by Example and The Rust Programming Language.
Assemble the dataset into the llama3 chat template
It is similar to the following. Each entry should be all in one line with linebreaks denoted as
\n
.Finetune
we will finetune based on the llama3-8b-instruct model.
You are free to use any finetune tools. But if you are unsure, we recommend using llama.cpp's
finetune
utility. See an example. It can run on CPUs. We will provide the computing resources required for the finetuing.Objective 2: Code QA model
Create a dataset with the following three fields
We are looking for at least 100 chapter + Q + A rows.
You could use ChatGPT to generate these questions and answers based on the chapter content.
Assemble the dataset into the llama3 chat template
It is similar to the following. Each entry should be all in one line with linebreaks denoted as \n.
Finetune
Due to the chapter-long context length used in this dataset, we will finetune based on the 262k long context length llama3-8b-instruct model.
You are free to use any finetune tools. But if you are unsure, we recommend using llama.cpp's finetune utility. See an example. It can run on CPUs. We will provide the computing resources required for the finetuing.
Objective 3: Compare the two finetuned models
Start the finetuned models using the LlamaEdge API server, and test them on commonly used scenarios.
LFX
Expected outcome: Two finetuned models based on Llama3-8b for Rust code review and QA.
Recommended skills:
Mentor:
Application link: ttps://mentorship.lfx.linuxfoundation.org/project/d52d172e-598d-4817-be97-3338d5b96432
Appendix
The text was updated successfully, but these errors were encountered: