These are chat archives for torch/torch7

13th
Jun 2015
xuwang
@xuwangyin
Jun 13 2015 00:06 UTC
Hi guys, I have a question about image data compression. I have a directory of images to be trained using neural networks. For some reason, i need to transfer the images from my local machine to the amazon ec2 machine. Since i have several gigabytes of images, i plan to use the command "tar -jcf" to compress the directory , but i am not sure that the images after decompression will be the same with the images before compression. My images are 32*32, grayscale, MNIST-like images. Any suggestion?
Javier
@JavierMares
Jun 13 2015 00:24 UTC
How can I change a tensor type, or copy the contents of say a bytetensor into a doubletensor?
Javier
@JavierMares
Jun 13 2015 00:48 UTC
My apologies, I have found how to do it.
Alfredo Canziani
@Atcold
Jun 13 2015 02:51 UTC
@xuwangyin, the argument j means you will perform a bzip2(1) compression, which is lossless. Therefore, what you will decompress will be the exact thing you compressed on your machine. Hope it helps.
@Sandy4321, to see images while debugging I usually use this debugger guide and exec image.savePNG(...).
xuwang
@xuwangyin
Jun 13 2015 03:24 UTC
@Atcold so cool, thanks!
Hokie23
@Hokie23
Jun 13 2015 14:44 UTC
Hi folks, I would like to say thanks to all for the helpful discussions.
I am training a deep and wide model for a classification task. All Of my Images are of people in different poses (64 by 64). I am getting a very high train accuracy(99) within 2 epochs but the test is not so good. Does anyone have any tips on hyperparameter tweaking? Or should I up my image size to 128 by 128?
Thank you!
Laurens van der Maaten
@lvdmaaten
Jun 13 2015 15:36 UTC
@Hokie23 Your model is likely overfitting. I would suggest you try out methods to regularize your model, for instance, by training on augmented data samples, adding L2 regularization on the weights, and adding dropout in the final layers of your model.
@Hokie23 Increasing the image size may improve the training and test error if you manage to regularize your model well, but as it increases the number of weights it will increase overfitting instead of decreasing it (meaning you will likely need stronger regularization).
Hokie23
@Hokie23
Jun 13 2015 15:38 UTC
@lvdmaaten thank you for the attention! :) I have jittered samples and dropouts in the fully connected layers.
How do I add LA regularization in the model file?
L2*
Laurens van der Maaten
@lvdmaaten
Jun 13 2015 15:42 UTC
@Hokie23 The optim package allows you to set a parameter weightdecay in the config table that performs L2 regularization. Are you sure that you don't have a bug and that your test samples come from the same distribution as your training samples? How many training samples do you have anyway?
Hokie23
@Hokie23
Jun 13 2015 15:47 UTC
@lvdmaaten I had about 50000 images I am doing a 40000 and 10000 split. The 40000 training samples have been jittered so I have ~360 training files. Should I use a higher learning rate?
~360k*
Yep it is the same poses in both sets, but with different people
Laurens van der Maaten
@lvdmaaten
Jun 13 2015 15:50 UTC
@Hokie23 Hmmm that should be sufficient to get some reasonable test performance. The learning rate seems all right: your learning tries to minimize training error and is apparently quite effective in doing so. So it must be very severe overfitting or (perhaps more likely) a bug in your error measurements / the way you feed test data to the model.
Hokie23
@Hokie23
Jun 13 2015 15:53 UTC
@lvdmaaten well, I will try the L2 regularizer and get back. Will have to wait another 6 hours haha
thanks again!
Andrew Tulloch
@ajtulloch
Jun 13 2015 17:12 UTC
Nice @lvdmaaten :)
Eugenio Culurciello
@culurciello
Jun 13 2015 17:18 UTC
@Hokie23 do you have poses in all different environments, diff people, diff clothes etc? If not, your net will learn the common background etc, and not learn the object you want. Also do you have the object centered and occupying some 80% or more area... to begin with that is a good first test. In other words, data matters a lot, if it is not collected well, nothing will work!
Ajay Talati
@AjayTalati
Jun 13 2015 17:28 UTC
@Hokie23 Is your dataset available to the public? I'd can try and have a go on it and see if I get similar results, as I'm doing something similar.
Hokie23
@Hokie23
Jun 13 2015 17:28 UTC
@culurciello I do have different lighting conditions, people, clothing etc but the same background .
@AjayTalati the dataset is a private sorry!
Is a private one*
The people are centered (with minor jitter)
Elad Hoffer
@eladhoffer
Jun 13 2015 21:13 UTC
@culurciello Just letting you know I updated my ImageNet example to use compressed jpgs in LMDB (~25GB footprint, with no speed loss) https://github.com/eladhoffer/ImageNet-Training
Eugenio Culurciello
@culurciello
Jun 13 2015 21:14 UTC
@Hokie23 if you have the same background.... then that is your problem, your network is learning that, not the data... you need diff bg
@eladhoffer thanks a lot for the info @Atcold @jhjin
@eladhoffer we are measuring loading time and we found ~1 sec for loading a batch of 128 jpegs of 256x256 from SSD. What is your #?
Elad Hoffer
@eladhoffer
Jun 13 2015 21:18 UTC
I get no slowdown in training, so must be lower than 4ms per sample, but I'll measure and let you know
Laurens van der Maaten
@lvdmaaten
Jun 13 2015 21:21 UTC
@culurciello @Hokie23 I don't understand? As long as the background doesn't co-vary with the class labels (different poses?), there shouldn't be much of a problem. Afaik a dataset like NORB has similar background everywhere, and can be classified quite well. The only way it may go wrong is if the background of the test samples differs from that of the training samples, that is, when training and test data are sampled from different distributions.
Elad Hoffer
@eladhoffer
Jun 13 2015 21:24 UTC
@culurciello ~3ms\sample for sequential read and ~12ms\sample for random access, on a non SSD disk. Including jpeg decompression
Hokie23
@Hokie23
Jun 13 2015 21:37 UTC
@lvdmaaten @culurciello the background is the inside of a car
people are driving and doing different actions. I have 2 different car interiors and 25 people
19 in train and 6 in test.
@lvdmaaten I am running my model with weight decay set in the sgd configuration
all the people have pictures from both backgrounds
John Armstrong
@jarmstrong2
Jun 13 2015 23:09 UTC
I'm wondering if anyone can give pointers on how to speed up learning with GPU's I am training RNNs with timesteps at most 1000. All my custom modules are purely tensor operations, no loops. I'm still finding the learning to be slow... Just interested if anyone has any hints or hacks. Thanks.
Eugenio Culurciello
@culurciello
Jun 13 2015 23:31 UTC
@Hokie23 two bg might not be enough, I would use a lot more for the task. Maybe 100 or more.
@lvdmaaten @Hokie23 we have trained networks for detecting people id in the past, in the same background. It failed miserably. Trust me... it is important to have data with same object but largely varying background poses, etc... you know the deal. Deep Learning is mostly in the data, once you have some decent training scripts (we do in Torch7, thank you all!)
@eladhoffer that is good and really fast. We need to use your lib then!
John Armstrong
@jarmstrong2
Jun 13 2015 23:49 UTC
should I be using cutorch or cunn for best results with GPU?