These are chat archives for deeplearning4j/deeplearning4j/earlyadopters

3rd
Mar 2018
Paul Dubs
@treo
Mar 03 12:56
I wonder, why is cudnnDropoutBackward is called that often on a network that doesn't have any dropout specified:
image.png
Paul Dubs
@treo
Mar 03 13:19
There's also something fishy going on with the Workspaces
they keep creeping up in size, until the learning crashes
The network I'm training only has 1569983 Parameters, so it is tiny, and the batches are only 5mb in size each, so I wouldn't expect that training such a network would require more than 11GB of GPU RAM
(That is all on todays snapshots)
Paul Dubs
@treo
Mar 03 13:58
It seems to be somehow related to mask arrays
Paul Dubs
@treo
Mar 03 14:04
Because once it reaches an equilibrium on an iterator without mask arrays
Paul Dubs
@treo
Mar 03 14:19
:confused: ok, seems it isn't related to mask arrays. Using the same iterator, but commenting the masking out, results in the same problem
raver119
@raver119
Mar 03 14:42
do you have something reproducible for us?
:)
Paul Dubs
@treo
Mar 03 14:50
Will create something... thought I may find a work around before I submit an issue
raver119
@raver119
Mar 03 14:52
there were quite a few changes to workspaces recently, but i hadn't heard of anything like you're describing
Paul Dubs
@treo
Mar 03 16:24
hm... it looks like I can't properly reproduce it and for some reason my original doesn't work anymore Oo I guess it is time for a pause
Paul Dubs
@treo
Mar 03 17:31
yay, fixed it :D the original problem was that my data set contained some too long sequences, and the other one was fixed with a fresh shapshot pull
Richard Corbishley
@rcorbish
Mar 03 22:29
I figured out issue: 2675. It's a limitation in the cuda svd
From nvidia: gesvd only supports m>=n ( http://docs.nvidia.com/cuda/cusolver/index.html#cuds-lt-t-gt-gesvd )
I'll see if we can put in a workaround, should be able to transpose & switch U & VT. Let me test some ideas
At a minimum we should throw a better exception