The padding token choice #1577

Closed Answered by hiyouga

JianhuiWei7 asked this question in Q&A

The padding token choice #1577

Nov 21, 2023

· 1 comments · 1 reply

Answered by hiyouga Return to top

JianhuiWei7
Nov 21, 2023

In the function "preprocess_packed_supervised_dataset", the code used -100 as the padding for ignoring.

Why are we using -100? can we use others?
And when the model sees -100 how does it ignore the token? and ignore the loss it produces.

Answered by hiyouga

https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

View full answer

Replies: 1 comment 1 reply

hiyouga
Nov 21, 2023
Maintainer

https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html

1 reply

JianhuiWei7 Nov 21, 2023
Author

明白了谢谢！

Answer selected by JianhuiWei7

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment