训练中途提示Thread 0x00004ba8 (most recent call first):

[testworksizan]

epoch 4/12
steps: 33%|██▌ | 6153/18768 [11:58:50<24:33:46, 7.01s/it, Average key norm=0.793, Keys Scaled=0, avr_loss=0.0832]Windows fatal exception: access violation

Current thread 0x00000bd4 (most recent call first):
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 178 in Load
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 244 in Load
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\event_file_loader.py", line 270 in Load
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\directory_watcher.py", line 118 in _LoadInternal
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\directory_watcher.py", line 88 in Load
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\plugin_event_accumulator.py", line 183 in Reload
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 239 in Worker
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\plugin_event_multiplexer.py", line 261 in Reload
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\backend\event_processing\data_ingester.py", line 108 in _reload
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\threading.py", line 953 in run
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\threading.py", line 1016 in _bootstrap_inner
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\threading.py", line 973 in _bootstrap

Thread 0x00004ba8 (most recent call first):
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\selectors.py", line 315 in _select
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\selectors.py", line 324 in select
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\socketserver.py", line 232 in serve_forever
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\werkzeug\serving.py", line 804 in serve_forever
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\program.py", line 295 in _run_serve_subcommand
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\program.py", line 276 in main
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\absl\app.py", line 254 in _run_main
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\absl\app.py", line 308 in run
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\main.py", line 46 in run_main
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\site-packages\tensorboard\main.py", line 53 in
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\runpy.py", line 86 in _run_code
File "C:\Users\songz\Documents\app\lora-scripts-v1.8.5\python\lib\runpy.py", line 196 in _run_module_as_main
steps: 33%|██▎ | 6256/18768 [12:13:36<24:27:13, 7.04s/it, Average key norm=0.794, Keys Scaled=329, avr_loss=0.083]
saving checkpoint: ./output\0407end-000004.safetensors