添加噪声微调模型

[sevenandseven]

I encountered the following error when trying to add noise parameters during fine-tuning of embeddings models. How should I resolve this?

The command I ran is:
CUDA_VISIBLE_DEVICES=7 torchrun --standalone --nnodes=1 --nproc_per_node 1 -m FlagEmbedding.baai_general_embedding.finetune.run
--output_dir ./results/v2.0/embedding_1epoch_noise5
--model_name_or_path /media/ai/Embedding/bge-m3
--train_data /media/ai/fine_data/query_aneg.jsonl
--learning_rate 1e-5
--fp16
--num_train_epochs 1
--per_device_train_batch_size 8
--gradient_accumulation_steps 32
--dataloader_drop_last True
--normlized True
--temperature 0.02
--query_max_len 64
--passage_max_len 256
--train_group_size 1
--logging_steps 1
--logging_strategy steps
--save_steps 100
--save_strategy steps
--neftune_noise_alpha 5
--save_total_limit 10

my error is:
Traceback (most recent call last):
File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/media/ai/HDD/Teamwork/wangenzhi/FlagEmbedding-master/official/FlagEmbedding/FlagEmbedding/baai_general_embedding/finetune/run.py", line 120, in
main()
File "/media/ai/HDD/Teamwork/wangenzhi/FlagEmbedding-master/official/FlagEmbedding/FlagEmbedding/baai_general_embedding/finetune/run.py", line 111, in main
trainer.train()
File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/site-packages/transformers/trainer.py", line 1558, in train
self.model = self._activate_neftune(self.model)
File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/site-packages/transformers/trainer.py", line 665, in _activate_neftune
embeddings = unwrapped_model.get_input_embeddings()
File "/home/ai/anaconda3/envs/langchain_chatchat/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1614, in getattr
raise AttributeError("'{}' object has no attribute '{}'".format(
AttributeError: 'BiEncoderModel' object has no attribute 'get_input_embeddings'
ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: 1) local_rank: 0 (pid: 10472) of binary: /home/ai/anaconda3/envs/langchain_chatchat/bin/python

[staoxiao]

@sevenandseven , you need to add function get_input_embeddings for BiEncoderModel class.

[sevenandseven]

@sevenandseven , you need to add function get_input_embeddings for BiEncoderModel class.

ok, thanks for you reply.