How to validate .wav file using custom finetuned IEMOCAP wav2vec2 model

[sdhungan]

For my thesis, i converted the wav2vec2 to a regression model to predict the valance and arousal. The model is trained and i have the folder "CKPT+2024-03-09+10-58-39+00" which contains various .ckpt files. I assume the model that i have trained is called model.ckpt (or is it wav2vec2.ckpt ???). Now the issue arises that i don't know how to load in the model to predict the float values from a recorded .wav file... I didn't find alot of documentation regarding this. The only thing i found was how to do it with the base model provided from hugginface, but this requires various files with parameters which i don't know how to ajust for my setup. If someone knows how to load in and evaluate a new .wav file, please let me know. This is the last step i need for my thesis before i can proceed to the next step.

Kind regards
Safal

[BenoitWang]

Hi @sdhungan, you'll need to declare a pretrainer class to load the model checkpoints.
First define a pretrainer in your .yaml file and then call the from_hparamsfunction to load your pretrained models and change the HF path to your local path. In your case you'll need both the audio encoder (wav2vec2.ckpt) and the regressor (model.ckpt). Please refer to this tuto as well as this HF example.

[sdhungan]

Thank you, i managed to get it working! The output of mine fine-tuned model has only 2 nodes instead of the original 4 nodes. When evaluating with a wav file i get 2 tensors with a value that is resonable and logical. But i do get the following userwarning:
UserWarning: Passing gradient_checkpointing to a config initialization is deprecated and will be removed in v5 Transformers. Using model.gradient_checkpointing_enable() instead, or if you are using the Trainer API, pass gradient_checkpointing=True in your TrainingArguments.
warnings.warn(
Some weights of Wav2Vec2Model were not initialized from the model checkpoint at facebook/wav2vec2-base and are newly initialized: ['wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original0', 'wav2vec2.encoder.pos_conv_embed.conv.parametrizations.weight.original1']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
speechbrain.lobes.models.huggingface_transformers.huggingface - Wav2Vec2Model is frozen.

Could it be that i loaded in the original Wav2Vec2Model with the 4 output nodes? Because my output of the model is as expected, i don't know if this error is something i need to investigate or not.
Kind regards
Safal

[BenoitWang]

Don't worry this is a known warning from huggingface's side and normally if you used the same output dimension for fine-tuning and inference it should be ok.