Model training outputs NaN weights on my custom drone dataset

[lwhhhh13]

I’m currently trying to train the model on my own drone dataset. However, I noticed that after training starts, the weights become NaN.
I’ve double-checked the data format and preprocessing steps to ensure they are consistent with the expected input, but the issue persists.
Is there any known condition under which the training might diverge and produce NaNs? Any advice would be greatly appreciated.
Thanks in advance!

[xjiangan]

Thanks for reporting this issue — and great to hear you're experimenting with your own drone dataset!

We haven’t encountered NaNs during training in our standard experiments, but one possible factor could be the depth distribution: aerial drone imagery often covers much larger depth ranges compared to ground images.

Here are a few suggestions to help debug and isolate the issue:

• Adjust depth-related hyperparameters.
• Start with a small subset of your data that covers only a small region to simplify debugging and rule out issues with large scenes.
• If possible, sharing a minimal processed dataset would help us reproduce and better understand the problem.
• Check and share TensorBoard logs — they may provide clues on where and when the training diverges.

Let us know how it goes — happy to assist further once we have a bit more information!

[jiang-junlong]

我也遇见了相同的问题，不过我的数据集是在室内拿手持设备录得，视角比较稀疏。
glace跑起来精度还行，ace精度跑起来精度有点低。

[xjiangan]

我也遇见了相同的问题，不过我的数据集是在室内拿手持设备录得，视角比较稀疏。 glace跑起来精度还行，ace精度跑起来精度有点低。

Thanks for the additional info — that’s very helpful.
It’s good to hear that GLACE runs with reasonable accuracy, which suggests your data format and preprocessing are likely correct.

I’m wondering about the details of your scene reconstruction:

• How was the reconstruction done?
• Does it have a metric scale, or is it in arbitrary SfM units?
• Are the depth-related hyperparameters adjusted to match the scale of your reconstruction?

If possible, could you share:

1. A minimal processed dataset that reproduces the NaN behavior.
2. The configuration you used for training when the issue occurred.

This will help us reproduce the issue and pinpoint the cause.

[2601705041]

相同的问题。 我在校园和室内房间两个场景都进行了训练，其中这张图片是室内房间训练的结果，可以看到在loss变小（about1.6）后会突然出现NaN 的行为，在校园大场景中同样会出现这个问题，只不过是在about9时就会出现。 同时关于配置问题，都遵循项目的处理与训练命令，用process_data.py 成功转换格式。对于房间数据集，scr-overlap-score --data data/aachen/train --max_depth 50，将max_depth 改为了5，其他遵循标准命令。

[2601705041]

相同的问题。 我在校园和室内房间两个场景都进行了训练，其中这张图片是室内房间训练的结果，可以看到在loss变小（about1.6）后会突然出现NaN 的行为，在校园大场景中同样会出现这个问题，只不过是在about9时就会出现。 同时关于配置问题，都遵循项目的处理与训练命令，用process_data.py 成功转换格式。对于房间数据集，scr-overlap-score --data data/aachen/train --max_depth 50，将max_depth 改为了5，其他遵循标准命令。

我认为关键在于学习率，当然对不同大小的场景scr-overlap-score --data data/j222/train --max_depth 5 ，max_depth 参数也会对效果有影响，但是针对无法训练的问题关键在于学习率，在我的室内场景中将optimizers.head.scheduler.learning_rate_max改小最终成功训练了。 thanks for Gemini
scr-train scrfacto --data data/lab-222 --pipeline.datamanager.train_dataset.feat_name pose_n2c.pt
--machine.num-devices 1
--mixed-precision True
--use-grad-scaler True
--optimizers.head.scheduler.learning_rate_max 0.0005

[xjiangan]

Hi @2601705041 , thank you very much for the detailed report, which is extremely helpful.

May I ask whether your dataset is reconstructed in metric scale, or if it uses arbitrary SfM units? If the scene scale differs significantly from the default setups we tested, some additional parameters may also need adjustment. In particular, you may want to check depth-related settings such as depth_target and the depth bias standard deviation db_std in ReproLossConfig, which can be sensitive to scale.

It’s great to hear that reducing the learning rate resolved the training instability. This aligns well with the intuition. Another related factor is the effective batch size: the default configuration is tuned for multi-GPU training (e.g., 4 GPUs). If you’re training on a single GPU, a common heuristic is to scale the learning rate linearly with batch size, e.g., reducing it to roughly 1/4 of the default value, which matches your observation.

Thanks again for sharing both the analysis and the working configuration, which will be very useful for other users running into similar issues.

[2601705041]

非常感谢你的回答。metric scale和arbitrary SfM units，我都进行过训练，不过我好像没有对ReproLossConfig的depth_target进行修改，ReproLossConfig在哪里？