Hi ! Here is Coobiw 👋

🙋‍♂️ About Me:

• 👨‍🦰 I’m currently a M.Phil candidate of Peking University.
• 👦 Before that, I received the (Honours) B.E., HUST.
• ❤️‍🔥 Now, I am intersted in Multi-modal Learning especially MLLM.
• 💥 In 2023 summer, I take part in OSPP(Open Source Promotion Plan) Summer Camp , with the honor of contributing for MMPretrain to build prompt-based classifier.
  • Now, the implement of zero-shot CLIP classifier has been merged to the main branch. PR Link
  • The implement of RAM(Recognize Anything Model) has been merged to the dev branch. Welcome to use the gradio WebUI to test it on MMPretrain! PR Link
• 💥 2023.10: I implement MiniGPT4Qwen, which is a toy model aligning MiniGPT4 with Qwen-Chat LLM model. I just use 18.8k high quality instruction-tuning data(bi-lingual, selected from minigpt4 and llava). Just fine-tuning the projection layer (3M trainable parameters), this model support Chinese and English! MiniGPT4Qwen
• 💥 2024.2: I extend MiniGPT4Qwen to MPP-Qwen14B(Multimodal Pipeline Parallel), scaling up both the LLM(to Qwen-14B-Chat) and pretrain-data(using LLaVA-pretrain-data). I also unfreeze the whole LLM during SFT-stage. All training is conducted on 3090/4090 GPUs. To prevent poverty (24GB of VRAM) from limiting imagination, I implemented an MLLM version based on deepspeed Pipeline Parallel. Pre-training can be completed in 22 hours on 2x4090s, while SFT requires training on 6x4090s (because it needs to fully activate the LLM), but due to the small amount of data, it only takes several hours.MPP-Qwen14B