Sure, happy to help!

Using the ResNetEncoder and ResNetDecoder models results in a pretty hefty (~7MB/s) memory leak on CPU memory, which ultimately results in training crashing after a couple days.

I traced it down to a few lines in the custom batch-norm. The variables self.stored_mean, self.stored_var seem to only be used during inference, but are not protected from computing gradients, and thus are extending the computation graph on each call to bn.forward().

An old issue on the main PyTorch repo seems to validate these findings: https://github.com/pytorch/pytorch/issues/20275

I've tested with this change, and the memory usage remains constant.

Hey -- thanks for releasing the code, it's been very useful.

I've been reading through to try to make I understand everything, and there's one part that has me a bit confused:

In the top-level training script, train.py, an instance of the main training model ZbufferModelPts is instantiated, which in turn instantiates all the sub-models, which include the depth estimator, the feature encoder, and the feature decoder.

In particular, the feature decoder is instantiated as: https://github.com/facebookresearch/synsin/blob/82ff948f91a779188c467922c8f5144018b40ac8/models/z_buffermodel.py#L35

which, if we've chosen to use the ResNet blocks (as is default in the parameters shown in train.sh), will return: https://github.com/facebookresearch/synsin/blob/82ff948f91a779188c467922c8f5144018b40ac8/models/networks/utilities.py#L31

Shouldn't the channels_in here be 64, since it's taking as input the re-projected features? Or am I missing something?

Good catch, thanks!

It's probably worth making a pull request at some point to change all the default parameters to those used in the paper...

Thanks!

Yes, you're right that the default parameter for get_decoder() is UNetDecoder64, but in train.sh the parameters specifically override this default:

https://github.com/facebookresearch/synsin/blob/82ff948f91a779188c467922c8f5144018b40ac8/train.sh#L11

Also, I've been following that codepath, because from my understanding of the paper, the ResNet architecture is the one used in the proposed method. From the paper:

Our spatial feature network and refinement networks are composed of ResNet blocks. (Appendix B)

It also mentions:

We experimented with using a UNet architecture instead of a sequence of ResNet blocks for the spatial feature network and refinement network. This led to much worse results and was more challenging to train. (Appendix E)

Before I dug into the code, I assumed that the default parameters (when directly calling train.py, or importing a ZBufferModel in another script) would have reproduced the architecture and parameters described in the paper, but it seems that this might not be the case.

Hey -- thanks for releasing the code, it's been very useful.

I've been reading through to try to make I understand everything, and there's one part that has me a bit confused:

In the top-level training script, train.py, an instance of the main training model ZbufferModelPts is instantiated, which in turn instantiates all the sub-models, which include the depth estimator, the feature encoder, and the feature decoder.

In particular, the feature decoder is instantiated as: https://github.com/facebookresearch/synsin/blob/82ff948f91a779188c467922c8f5144018b40ac8/models/z_buffermodel.py#L35

which, if we've chosen to use the ResNet blocks (as is default in the parameters shown in train.sh), will return: https://github.com/facebookresearch/synsin/blob/82ff948f91a779188c467922c8f5144018b40ac8/models/networks/utilities.py#L31

Shouldn't the channels_in here be 64, since it's taking as input the re-projected features? Or am I missing something?