训练异常生成模块，最后崩掉了

[DqqHns]

想问一下大家有没有遇过这种情况，训练异常生成模块，到最后才崩掉，训练了6天：Epoch 428: 100%|█| 2331/2331 [18:55<00:00, 2.05it/s, loss=0.00671, v_num=0, train/loss_simple_step=0.00161, train/loss_vlb_step=5.32e-6, train/loss_step=0.00161, global_Average Epoch time: 1135.78 seconds
Average Peak memory 17159.02MiB
Epoch 429: 0%| | 0/2331 [00:00<?, ?it/s, loss=0.00671, v_num=0, train/loss_simple_step=0.00161, train/loss_vlb_step=5.32e-6, train/loss_step=0.00161, global_step=1e+6, thuggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using tokenizers before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Epoch 429: 18%|▏| 430/2331 [03:13<14:16, 2.22it/s, loss=0.00793, v_num=0, train/loss_simple_step=0.00195, train/loss_vlb_step=6.72e-6, train/loss_step=0.00195, global_stEpoch 429, global step 999999: val/loss_simple_ema was not in top 1
Average Epoch time: 197.85 seconds
Average Peak memory 17159.02MiB
Epoch 429: 18%|▏| 430/2331 [03:18<14:36, 2.17it/s, loss=0.00793, v_num=0, train/loss_simple_step=0.00195, train/loss_vlb_step=6.72e-6, train/loss_step=0.00195, global_st
Saving latest checkpoint...

Traceback (most recent call last):
File "main.py", line 871, in
trainer.test(model, data)
File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 904, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _test_impl
results = self._run(model, ckpt_path=self.tested_ckpt_path)
File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1115, in _run
verify_loop_configurations(self, model)
File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 38, in verify_loop_configurations
__verify_eval_loop_configuration(trainer, model, "test")
File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 182, in __verify_eval_loop_configuration
raise MisconfigurationException(f"No {loader_name}() method defined to run Trainer.{trainer_method}.")
pytorch_lightning.utilities.exceptions.MisconfigurationException: No test_dataloader() method defined to run Trainer.test.
(Anomalydiffusion) root@autodl-container-857740b613-a811c0e1:~/autodl-tmp/anomalydiffusion-master#

[bye111]

请问您解决了吗，我也遇到相同情况，应该不是数据集问题：
Traceback (most recent call last):
File "train_mask.py", line 915, in
trainer.test(model, data)
File "D:\anaconda3\envs\stylegan3\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 904, in test
return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule)
File "D:\anaconda3\envs\stylegan3\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 682, in _call_and_handle_interrupt
return trainer_fn(*args, **kwargs)
File "D:\anaconda3\envs\stylegan3\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 947, in _test_impl
results = self._run(model, ckpt_path=self.tested_ckpt_path)
File "D:\anaconda3\envs\stylegan3\lib\site-packages\pytorch_lightning\trainer\trainer.py", line 1115, in _run
verify_loop_configurations(self, model)
File "D:\anaconda3\envs\stylegan3\lib\site-packages\pytorch_lightning\trainer\configuration_validator.py", line 38, in verify_loop_configurations
__verify_eval_loop_configuration(trainer, model, "test")
File "D:\anaconda3\envs\stylegan3\lib\site-packages\pytorch_lightning\trainer\configuration_validator.py", line 182, in __verify_eval_loop_configuration
raise MisconfigurationException(f"No {loader_name}() method defined to run Trainer.{trainer_method}.")
pytorch_lightning.utilities.exceptions.MisconfigurationException: No test_dataloader() method defined to run Trainer.test.

[wenbolics]

想问一下大家有没有遇过这种情况，训练异常生成模块，到最后才崩掉，训练了6天：Epoch 428: 100%|█| 2331/2331 [18:55<00:00, 2.05it/s, loss=0.00671, v_num=0, train/loss_simple_step=0.00161, train/loss_vlb_step=5.32e-6, train/loss_step=0.00161, global_Average Epoch time: 1135.78 seconds Average Peak memory 17159.02MiB Epoch 429: 0%| | 0/2331 [00:00<?, ?it/s, loss=0.00671, v_num=0, train/loss_simple_step=0.00161, train/loss_vlb_step=5.32e-6, train/loss_step=0.00161, global_step=1e+6, thuggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) Epoch 429: 18%|▏| 430/2331 [03:13<14:16, 2.22it/s, loss=0.00793, v_num=0, train/loss_simple_step=0.00195, train/loss_vlb_step=6.72e-6, train/loss_step=0.00195, global_stEpoch 429, global step 999999: val/loss_simple_ema was not in top 1 Average Epoch time: 197.85 seconds Average Peak memory 17159.02MiB Epoch 429: 18%|▏| 430/2331 [03:18<14:36, 2.17it/s, loss=0.00793, v_num=0, train/loss_simple_step=0.00195, train/loss_vlb_step=6.72e-6, train/loss_step=0.00195, global_st Saving latest checkpoint...

Traceback (most recent call last): File "main.py", line 871, in trainer.test(model, data) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 904, in test return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _test_impl results = self._run(model, ckpt_path=self.tested_ckpt_path) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1115, in _run verify_loop_configurations(self, model) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 38, in verify_loop_configurations __verify_eval_loop_configuration(trainer, model, "test") File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 182, in __verify_eval_loop_configuration raise MisconfigurationException(f"No {loader_name}() method defined to run Trainer.{trainer_method}.") pytorch_lightning.utilities.exceptions.MisconfigurationException: No test_dataloader() method defined to run Trainer.test. (Anomalydiffusion) root@autodl-container-857740b613-a811c0e1:~/autodl-tmp/anomalydiffusion-master#

请问大佬解决了吗？

[DqqHns]

没，我直接不复现了，直接跑

…

---原始邮件--- 发件人: ***@***.***&gt; 发送时间: 2025年8月20日(周三) 下午3:23 收件人: ***@***.***&gt;; 抄送: ***@***.******@***.***&gt;; 主题: Re: [sjtuplayer/anomalydiffusion] 训练异常生成模块，最后崩掉了 (Issue #112) wenbolics left a comment (sjtuplayer/anomalydiffusion#112) 想问一下大家有没有遇过这种情况，训练异常生成模块，到最后才崩掉，训练了6天：Epoch 428: 100%|█| 2331/2331 [18:55<00:00, 2.05it/s, loss=0.00671, v_num=0, train/loss_simple_step=0.00161, train/loss_vlb_step=5.32e-6, train/loss_step=0.00161, global_Average Epoch time: 1135.78 seconds Average Peak memory 17159.02MiB Epoch 429: 0%| | 0/2331 [00:00<?, ?it/s, loss=0.00671, v_num=0, train/loss_simple_step=0.00161, train/loss_vlb_step=5.32e-6, train/loss_step=0.00161, global_step=1e+6, thuggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... To disable this warning, you can either: - Avoid using tokenizers before the fork if possible - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) Epoch 429: 18%|▏| 430/2331 [03:13<14:16, 2.22it/s, loss=0.00793, v_num=0, train/loss_simple_step=0.00195, train/loss_vlb_step=6.72e-6, train/loss_step=0.00195, global_stEpoch 429, global step 999999: val/loss_simple_ema was not in top 1 Average Epoch time: 197.85 seconds Average Peak memory 17159.02MiB Epoch 429: 18%|▏| 430/2331 [03:18<14:36, 2.17it/s, loss=0.00793, v_num=0, train/loss_simple_step=0.00195, train/loss_vlb_step=6.72e-6, train/loss_step=0.00195, global_st Saving latest checkpoint... Traceback (most recent call last): File "main.py", line 871, in trainer.test(model, data) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 904, in test return self._call_and_handle_interrupt(self._test_impl, model, dataloaders, ckpt_path, verbose, datamodule) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 682, in _call_and_handle_interrupt return trainer_fn(*args, **kwargs) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 947, in _test_impl results = self._run(model, ckpt_path=self.tested_ckpt_path) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/trainer.py", line 1115, in _run verify_loop_configurations(self, model) File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 38, in verify_loop_configurations __verify_eval_loop_configuration(trainer, model, "test") File "/root/miniconda3/envs/Anomalydiffusion/lib/python3.8/site-packages/pytorch_lightning/trainer/configuration_validator.py", line 182, in __verify_eval_loop_configuration raise MisconfigurationException(f"No {loader_name}() method defined to run Trainer.{trainer_method}.") pytorch_lightning.utilities.exceptions.MisconfigurationException: No test_dataloader() method defined to run Trainer.test. (Anomalydiffusion) ***@***.***:~/autodl-tmp/anomalydiffusion-master# 请问大佬解决了吗？ — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[hyao1]

请问解决了吗，看起来是没有设置测试集，忽略掉也可以跑，不影响最后的模型训练

[wenbolics]

没解决，但是作者说没事

…

---原始邮件--- 发件人: "Hang ***@***.***&gt; 发送时间: 2025年8月27日(周三) 晚上6:33 收件人: ***@***.***&gt;; 抄送: ***@***.******@***.***&gt;; 主题: Re: [sjtuplayer/anomalydiffusion] 训练异常生成模块，最后崩掉了 (Issue #112) hyao1 left a comment (sjtuplayer/anomalydiffusion#112) 请问解决了吗，看起来是没有设置测试集，忽略掉也可以跑，不影响最后的模型训练 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: ***@***.***&gt;