[observability] OTEL Trace/Event for training rendezvous, gpu check, flash checkpoint, etc.

[liyzcj]

In the process of training large-scale distributed models, we often encounter a variety of issues. Currently, we can only analyze the possible causes of these issues by examining the logs. To enhance the observability of these operations and support automated diagnostic capabilities, we aim to integrate standard Trace functionality.

For example: