RuntimeError: probability tensor contains either inf, nan or element < 0

[dongzhiwen1218]

I got this error while doing Fine-tuning:

File "/mnt/oss-data/xxx/minillm/transformers/src/transformers/generation/utils.py", line 3000, in sample
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either inf, nan or element < 0
next_tokens = torch.multinomial(probs, num_samples=1).squeeze(1)
RuntimeError: probability tensor contains either inf, nan or element < 0
[I ProcessGroupNCCL.cpp:844] [Rank 2] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 1] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 3] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 0] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 2] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 0] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 3] NCCL watchdog thread terminated normally
[I ProcessGroupNCCL.cpp:844] [Rank 1] NCCL watchdog thread terminated normally
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 12080 closing signal SIGTERM
WARNING:torch.distributed.elastic.multiprocessing.api:Sending process 12081 closing signal SIGTERM
...
Does anyone see the same issue?

model config:
{
"_name_or_path": "/xxx",
"architectures": [
"LlamaForCausalLM"
],
"bos_token_id": 1,
"eos_token_id": 2,
"hidden_act": "silu",
"hidden_size": 5120,
"initializer_range": 0.02,
"intermediate_size": 13824,
"max_position_embeddings": 4096,
"model_type": "llama",
"num_attention_heads": 40,
"num_hidden_layers": 40,
"num_key_value_heads": 40,
"pad_token_id": 0,
"pretraining_tp": 1,
"rms_norm_eps": 1e-05,
"rope_scaling": null,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.31.0",
"use_cache": true,
"vocab_size": 55296
}

cmd:
export NCCL_DEBUG=""
export WANDB_DISABLED=True
export TF_CPP_MIN_LOG_LEVEL=0
export TORCH_CPP_LOG_LEVEL=0
torchrun --nproc_per_node 4 --nnodes 1 --node_rank 0 --master_addr localhost --master_port 2012 /mnt/oss-data/xxx/minillm/finetune.py --base-path /mnt/oss-data/xxx/minillm --model-path /mnt/oss-data/xxx/minillm/xxx/ --ckpt-name xxx --n-gpu 4 --model-type llama2 --gradient-checkpointing --model-parallel --model-parallel-size 4 --data-dir /mnt/oss-data/xxx/minillm/processed_data/dolly/full/llama2/ --num-workers 0 --dev-num 500 --lr 0.00001 --batch-size 4 --eval-batch-size 8 --gradient-accumulation-steps 2 --warmup-iters 0 --lr-decay-style cosine --weight-decay 1e-2 --clip-grad 1.0 --epochs 10 --max-length 512 --max-prompt-length 256 --do-train --do-valid --eval-gen --save-interval -1 --eval-interval -1 --log-interval 4 --mid-log-num 1 --save /mnt/oss-data/xxx/minillm/results/llama2/train/sft --seed 10 --seed-order 10 --deepspeed --deepspeed_config /mnt/oss-data/xxx/minillm/configs/deepspeed/ds_config_zero2_offload.json --type lm --do-sample --top-k 1 --top-p 0.9 --temperature 1.0