Name		Name	Last commit message	Last commit date
parent directory ..
images		images
tiny-scripts		tiny-scripts
NicerTrace.py		NicerTrace.py
README.md		README.md
make-tiny-models-tokenizers-datasets.md		make-tiny-models-tokenizers-datasets.md
nccl-performance-debug.md		nccl-performance-debug.md
pytorch.md		pytorch.md
tools.md		tools.md
torch-distributed-gpu-test.py		torch-distributed-gpu-test.py
torch-distributed-hanging-solutions.md		torch-distributed-hanging-solutions.md
underflow_overflow.md		underflow_overflow.md
underflow_overflow.py		underflow_overflow.py

README.md

Debugging and Troubleshooting

Guides

Debugging PyTorch programs
Diagnosing Hangings and Deadlocks in Multi-Node Multi-GPU Python Programs
Troubleshooting NVIDIA GPUs
Underflow and Overflow Detection
NCCL Debug and Performance - notes for debugging NCCL-based software and tuning it up for the peak performance

Tools

Debug Tools
torch-distributed-gpu-test.py - this a torch.distributed diagnostics script that checks that all GPUs in the cluster (one or many nodes) can talk to each other and allocate gpu memory.
NicerTrace - this is an improved trace python module with multiple additional flags added to the constructor and more useful output.