Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 31 2019 20:13
  • Jan 31 2019 07:16
    nuczzz starred torch/torch7
  • Jan 31 2019 03:36
    ajhool commented #1193
  • Jan 31 2019 03:21
    ajhool closed #1192
  • Jan 31 2019 03:21
    ajhool commented #1192
  • Jan 30 2019 17:05
    marwon34 commented #1146
  • Jan 30 2019 13:01
    elvis1423 commented #1180
  • Jan 30 2019 13:00
    elvis1423 commented #1180
  • Jan 30 2019 02:02
    EternalSunLhx starred torch/torch7
  • Jan 29 2019 11:07
    berlam starred torch/torch7
  • Jan 28 2019 21:20
    generic-github-user starred torch/torch7
  • Jan 28 2019 20:01
  • Jan 28 2019 20:00
    chipper1 starred torch/torch7
  • Jan 28 2019 13:36
  • Jan 28 2019 09:43
    jkadowaki starred torch/torch7
  • Jan 28 2019 08:28
  • Jan 28 2019 02:58
    pavq forked
    pavq/torch7
  • Jan 26 2019 12:39
    stu17682 commented #1192
  • Jan 26 2019 08:33
    ajhool opened #1193
  • Jan 26 2019 05:08
    ajhool opened #1192
Sam Stites
@stites
oh! ahah, sorry
Marek Cerny
@marek094
The core of the question — is it possible easily switch from one GPU to 8GPU without much efford?
I guess TensorFlow can easily use all hardware (CPUs, GPUs) together on his own.
Sam Stites
@stites
In lua-torch? I am not too sure
Marek Cerny
@marek094
:'(
ok
Sam Stites
@stites
My main recommendation I would checking out using asynchronous cuda-based tensors
I think that's the primary mechanism that gets you speedups for multigpu. Others might have better tips and tricks (it's just usually quiet here, so I try to be the first responder).
Sam Stites
@stites
Yeah, because torch dynamically constructs the graph, I'm not sure it can confirm that two tensors will be on the same card for an op (and torch doesn't have a management layer for this). You would want to use something like the Tensor Comprehensions library for taht
or you would have to manually coordinate how the tensors are merged (which shouldn't be too hard).
I know Tensor Comprehensions are an external library (with PyTorch integration), you can check out the whitepaper here: https://research.fb.com/announcing-tensor-comprehensions/
Marek Cerny
@marek094
thanks you, but it seems like more work than wait cca 4 times longer :D
Sam Stites
@stites
@marek094 Maybe it is actually as simple as: bamos/densenet.pytorch#5
adamw005
@adamw005
Any advice on trouble shooting an nn model? It's not learning and guessing around the same value every time
Sam Stites
@stites
Unit test with fixed seeds and prefilled tensors for a step or two
You may also get to rollout half the epoch or episode (depending on your domain, episodes only if you can adjust the environment) and then unit test some more.
Sam Stites
@stites
There are a few posts scattered across the internet which might have more tips. In general stepping through your code with a debugger comes first, unit tests are just to codify that.
Also, keep a daily journal of your assumptions and actions. It helps with logic bugs and will keep you sane
Haha, also "log everything" and "chart everything" depending on how crazy things get
tastyminerals
@tastyminerals
@adamw005 if it doesn't learn it will predict same thing more or less all the time. Debugging a model is a tedious process since it can be a small bug or whole model architecture incorrectly chosen and applied to your problem space. Start with data preprocessing and make sure you do it right, check inputs and whether they align with your target values. Then double check your loss function, is it the right one for your task? Most of the time it is likely a preprocessing bug or wrong loss function application.
tastyminerals
@tastyminerals
@koen-dejonghe what's working in Scorch and what's not? In order to use our Torch models in production we had to rewrite them in Tensorflow and use its Java API to load the models in Scala. I am very interested in Scorch and Numsca but it looks like these are one-man projects for now. Are these just your toy projects?
adamw005
@adamw005
Thanks @tastyminerals
Binesh Bannerjee
@bnsh
@tastyminerals Oh that's funny re: rewriting in tensorflow/java... I thought I was the only one...
(I still prefer torch (though pyTorch these days...) But, it's hard to argue with tensorflow's deployment ability..)
tastyminerals
@tastyminerals
@bnsh rewriting is a pain, tensorflow models exhibit different behaviour with the same hypeparams and you basically need to redo your hyperparameters tuning all over. Besides we got into the situation where TF model refused to train unless you l2 normalized the input unlike in Torch. Minesweeping these small differences is a real quest.
Binesh Bannerjee
@bnsh
Oh I guess I'm a little lucky.. My deployed models don't need to be trained in the field... So, I just train them on pyTorch, and then copy the weights to my re-written tensorflow models... So I had to rewrite a few to make them pytorch friendly... I think bidirectionalLSTM was one, which had subtle differences between TF's implementation and Torch's... But, man development on pyTorch is just so much easier all this is still worth it...
Koen Dejonghe
@koen-dejonghe
@tastyminerals Thank you for your interest. Scorch and Numsca are one man projects. Of course you're welcome to contribute. Everyday components, from Linear, CNN and RNN layers, SGD, Nesterov and Adam optimizers, to Softmax and CrossEntropy criterions have been implemented. I also implemented a module that allows data parallelization. Almost everything is implemented in pure Scala, and tested with gradient checks, so should be correct. However, it's slow. As the backend of Numsca, I am using ND4j, and I find this to be really slow.
That is why I am trying to develop the same thing over, this time with the ATen library. You can take a peek here: https://github.com/koen-dejonghe/numsca-torch
Koen Dejonghe
@koen-dejonghe
The general outline is more or less ready (apart from tensor typing), and it seems to work much, much faster. I'm running into the occasional and inexplicable memory error (I think). I'm just about to implement convolutions.
Any advice is greatly appreciated.
tastyminerals
@tastyminerals

Oh I guess I'm a little lucky.. My deployed models don't need to be trained in the field... So, I just train them on pyTorch, and then copy the weights to my re-written tensorflow models... So I had to rewrite a few to make them pytorch friendly... I think bidirectionalLSTM was one, which had subtle differences between TF's implementation and Torch's... But, man development on pyTorch is just so much easier all this is still worth it...

That might not always work unfortunately since there are still TF-specific things like feature normalization which makes concrete TF model work while Torch doesn't care much.

@koen-dejonghe thumbs up for your efforts, your project helps out a lot for a Scala newbie like me. I only heard good things about ND4j, from DL4j devs :) Hmm, slow performance is an issue. Have you tried to figure it out in DL4j gitter channel? The devs are very active and could help there.
Koen Dejonghe
@koen-dejonghe
I've tried their channel.
adamw005
@adamw005
Looks really interesting, thanks @koen-dejonghe
adamw005
@adamw005
I was able to install tensorboardX and use it on non-gpu ec2 instances.. but doesn't seem to work on p2
I've installed it under pytorch_p36 environment successfully, but Jupyter notebook shows conda_pytorch_p36 as the environment - is this the issue? all environment have the conda prefix in jupyter but not in the terminal
adamw005
@adamw005
Also, is F.normalize() just taking each value and dividing by the max? Is there a normalization that subtracts the mean and divides by st dev?
adamw005
@adamw005
When I normalize both my x and y data I am able to get a perfect fit (1 hidden layer, 24 nodes). But when I don't normalize I just get a straight line (same model) - anyone know why that is?
image.png
Koen Dejonghe
@koen-dejonghe
What I find particularly interesting are the common pitfalls: An important point to make about the preprocessing is that any preprocessing statistics (e.g. the data mean) must only be computed on the training data, and then applied to the validation / test data. E.g. computing the mean and subtracting it from every image across the entire dataset and then splitting the data into train/val/test splits would be a mistake. Instead, the mean must be computed only over the training data and then subtracted equally from all splits (train/val/test).
adamw005
@adamw005
This is great - thanks @koen-dejonghe
adamw005
@adamw005
Is there a way to only pull a certain class from the STL10 dataset?
Robert Burke
@sharpobject
is there fast way to copy values from a table to a tensor? writing individual values to a tensor or the underlying DoubleStorage is way slower than writing to a table. So far the fastest way I found is to reuse some tables and make new tensors all the time using the constructor that takes a table.
it would be sweet to not make new tensors all the time though
Jean Senellart
@jsenellart
Cuda or regular tensor?
DenisDsh
@DenisDsh

Hello everybody,

I’m new to Torch and I’m using it for my master thesis project.
I am trying to integrate to an existing implementation a personalized loss function like below :

sum_loss = lambda * coral_loss + classification_loss

where coral_loss is an unsupervised loss that measures the “distance” between a source and a target batch.
Basically I want to minimize the classification loss on the source data while jointly minimizing this distance between source and target data distributions.

In PyTorch that is very easy since I can just call sum_loss.backward() to compute the gradients. However in Torch there are predefined Criterions in which I can only call backward(input,target) to compute the gradients of the loss function associated to that specific criterion.

How can I implement the new loss function and backpropagate to compute the gradients?

This is the loss that I would like to implement in Torch :

def CORAL(source, target):

   d = source.data.shape[1]

   # source covariance
   xm = torch.mean(source, 0, keepdim=True) - source
   xc = xm.t() @ xm

   # target covariance
   xmt = torch.mean(target, 0, keepdim=True) - target
   xct = xmt.t() @ xmt

   # frobenius norm between source and target
   loss = torch.mean(torch.mul((xc - xct), (xc - xct)))
   loss = loss/(4*d*d)

   return loss
Robert Burke
@sharpobject
@jsenellart regular tensor
Robert Burke
@sharpobject
DenisDsh
@DenisDsh
@sharpobject will take a look to that, thanks