RLHF for classification tasks

[vinodrajendran001]

I am trying to apply RLHF on a text classification task. You can imagine the text classification model i.e. policy model here is emotion classification. The pretrained model can output class numbers ranging between 1 and 10. The reward model should train with the dataset labelled with correct class numbers (assuming it is available). Finally, I want to optimize the policy model with reward model using PPO.

Can this be done with this library? If so, please help by illustrating the steps.

Thanks

[hijkzzz]

This should require quite a lot of modifications.
I suggest you read the train_ppo.py code carefully.

[openllmai0]

Here are two suggestions: 1. Text classification may not require complex RL algorithms. 2. If using RLHF, consider changing the output to label text instead of label number.