occur inconsistency of rendering quality after resuming a snapshot from file

[Karbo123]

Thanks for this great work! I'd like to report a small bug here. It would be great if you can provide some suggestions about it. I've tried to read and debug the codes, but I find it difficult to solve the problem because it's written in cpp and cuda.

Problem/Bug Description
I have observed an inconsistent behaviour related to the saved snapshot. The PSNR doesn't match. The PSNR just after the training (not saved to the disk yet) is reported as 35.7 or so, but the PSNR after resuming the snapshot becomes to just about 31.7.
I think this is because there are some things that are not saved to the file. I've tried the latest pull requests, but it can't solved the problem either. My NGP is built from the latest commit (ae2afc8).

To reproduce

# for training, run
CUDA_VISIBLE_DEVICES=$CUDA_CARD python ./scripts/run.py --mode nerf \
--scene $DATASET_DIR/nerf_synthetic/lego/transforms_train.json \
--test_transforms $DATASET_DIR/nerf_synthetic/lego/transforms_test.json \
--save_snapshot $NGP_RUN_SAVE_DIR/model.msgpack \
--screenshot_transforms $DATASET_DIR/nerf_synthetic/lego/transforms_test.json \
--screenshot_dir $NGP_RUN_SAVE_DIR \
--width 800 --height 800

# for testing, run
CUDA_VISIBLE_DEVICES=$CUDA_CARD python ./scripts/run.py --mode nerf \
--scene $DATASET_DIR/nerf_synthetic/lego/transforms_train.json \
--test_transforms $DATASET_DIR/nerf_synthetic/lego/transforms_test.json \
--load_snapshot $NGP_RUN_SAVE_DIR/model.msgpack \
--screenshot_transforms $DATASET_DIR/nerf_synthetic/lego/transforms_test.json \
--screenshot_dir $NGP_RUN_SAVE_DIR \
--width 800 --height 800

[Tom94]

Hi there, could you try passing --n_steps 0 to that second command? Otherwise, I'm afraid the second command will simply re-train the model, but using the loaded snapshot as initial starting point, which may lead to degenerate behavior.

[Karbo123]

Thank you very much! That really solved my problem!
It's actually not a bug, but it would be better if the demo script scripts/run.py could elaborate more on this.