[Kenichi Maehashi, chainer] I thought that overhead of copying
b from NumPy array to iDeep optimized array happens every time running linear, because b is not initialized as mdarray. I thought such overhead can be eliminated by initializing b as mdarray. https://chainer.slack.com/archives/C4MQ9RMNG/p1521094728000070
[fengyuan, chainer] Yes, you are right. That will be an optimization to reduce overhead of memory copy.
BTW, our design is not fully corresponding to GPU. Overhead about copy between CPU and GPU is considerable. According to our design in the very beginning, the most reason why weight is initialized in mdarray is to get a array in weight format. That will bring performance improvement to iDeep primitive computation. At that time, we have not taken 'b' into account, not taken memory copy into account.
BTW, our design is not fully corresponding to GPU. Overhead about copy between CPU and GPU is considerable. According to our design in the very beginning, the most reason why weight is initialized in mdarray is to get a array in weight format. That will bring performance improvement to iDeep primitive computation. At that time, we have not taken 'b' into account, not taken memory copy into account.
[Kenichi Maehashi, chainer] Thank you for the detailed explanation! I understood the policy of the original design.
So, it is OK to change
https://chainer.slack.com/archives/C4MQ9RMNG/p1525777655000192?thread_ts=1522843389.000152&cid=C4MQ9RMNG
So, it is OK to change
all_ready to accept (1, 2, 4) instead of (2, 4).https://chainer.slack.com/archives/C4MQ9RMNG/p1525777655000192?thread_ts=1522843389.000152&cid=C4MQ9RMNG
[fengyuan, chainer] In our design,
all_readyis a programmer helper function. The purpose is to check inputs whether could be accepted by current iDeep primitive or not, not to check whether dimension is valid for iDeepmdarrayor not. So, it is no needed to checkall_ready, whenWorbis initialized. Here is a mistake in Intel-Chainer release_v3. https://github.com/intel/chainer/blob/release_v3/chainer/links/connection/linear.py#L128- Programmer must aware valid/supported dimension of inputs in current iDeep primitive. By default, it is
(2, 4). Programmer could adjustsupported_ndimofall_readyaccordingly.
My suggestion here is to initializebinmdarraydirectly withoutall_ready. What do you think of that?
[fengyuan, chainer] e.g. Explicitly define supported_ndim, if an iDeep primitive just supports 1-D inputs.
if intel64.inputs_all_ready((x, y, z, ), supported_ndim = (1, )):
intel64.ideep.some_primitive(x, y ,z)
[Kenichi Maehashi, chainer] Thank you, now I understand why I got wrong. Chainer uses
https://github.com/chainer/chainer/blob/v5.0.0a1/chainer/variable.py#L770-L786
https://github.com/intel/chainer/blob/release_v3/chainer/variable.py#L728
all_ready in Variable.to_intel64, whereas intel-chainer does not use it in Variable.to_ia.https://github.com/chainer/chainer/blob/v5.0.0a1/chainer/variable.py#L770-L786
https://github.com/intel/chainer/blob/release_v3/chainer/variable.py#L728
[Kenichi Maehashi, chainer] I also hope this fix is included into iDeep4py 2.0. https://chainer.slack.com/archives/C4MQ9RMNG/p1517983472000240
Currently users are forced to install NumPy==1.13.0 and cannot use other versions of NumPy.
Setting requirements to
Currently users are forced to install NumPy==1.13.0 and cannot use other versions of NumPy.
Setting requirements to
numpy>=1.13.0 would be nice.
[Cao Zhong, chainer] It is nearly done. In master branch. We are deciding when to release it and what is in it. @kmaehashi
[Cao Zhong, chainer] Yes, we’ll remove that constraint.
[Kenichi Maehashi, chainer] Nice, thank you!
[Kenichi Maehashi, chainer] I’d like to copy data from NumPy array to iDeep array, and found
ideep4py.basic_copyto(dst, src) which seems like an equivalent for numpy.copyto(dst, src). Is this basic_copyto considered stable? Can we use this interface? (context: chainer/chainer#5009)
[Cao Zhong, chainer] @feng1.yuan
[Kenichi Maehashi, chainer] Ok, thanks for clarification!
[mingxiao huang, chainer] @mitmul ok, Chainermn's batch normalization implementation is heavily based on Chainer's batch normalization old version, I am trying to sync Chainermn's batch normalization with Chainer's batch normalization latest version. I will keep you informed with the progress. Thanks.
[Shunta Saito, chainer] Great! Thanks for letting us know it.
[mingxiao huang, chainer] @nishino hello, nishino, recently we trained resnet50 GPU model on single node and only got 73+% validation accuracy on ILSVRC2012, there is 2% gap as compared to SOTA. We used poly learning rate policy. It is said that you can got SOTA for resnet50 on your site, may you please share the hyper parameters with us? Thanks.
[mingxiao huang, chainer] @mitmul from http://on-demand.gputechconf.com/gtc/2018/presentation/s8889-training-imagenet-in-15-minutes-with-chainermn-a-scalable-distributed-dl-framework.pdf, we can see that you got a comparable accuracy of 74.9% on multi-node. May you please send the training script file you used to us? If you have ever trained resnet50 on single node, it would be better to send us the training script file you used for single node to us. Thanks.
[Shunta Saito, chainer] @kfukuda Hi Fukuda-san, he is mentioning your slides you presented at GTC and asking the training setting to achieve 74.9% accuracy on ImageNet with ResNet50. Can we share the training script with him?
[Keisuke Fukuda, chainer] I think we need to consult Akiba-san.
[Keisuke Fukuda, chainer] However, basically, everything is written in the paper
[Shunta Saito, chainer] Yeah, @mingxiao.huang Have you already read this paper? https://arxiv.org/abs/1711.04325
[Shunta Saito, chainer] What is the global minibatch size? I think the techniques written in the paper is for "Extremely large minibatch" as you can see in the title. If the global minibatch size is not so large because you are just using a single GPU, the technique required to achieve good results will be different. What do you think @kfukuda?
[mingxiao huang, chainer] @mitmul Yes, that is also my concern. for single node, we used batchsize=128. The technique required to achieve good results will be different for single node training, so, it would be nice if you have such a training script file for single node and send it to us. Thanks.
[Shunta Saito, chainer] Regarding a training script, I think it's OK to use the normal one for ResNet50 training on ImageNet-1K with 128 minibatchsize. You can find the example code in Chainer repository. Have you already tried that?
[mingxiao huang, chainer] @mitmul the example code in Chainer repository is rather simple, actually, we would like to reproduce your result with detailed hyper parameters you used, since we could not get SOTA accuracy on resnet50 on chainer at all, however, with same parameters, we can achieve SOTA on caffe.
[Shunta Saito, chainer] @mingxiao.huang What we intended to say was that you need to use "extremely large batchsize" to reproduce the result on paper. If you want to achieve over 74% accuracy by using only 1 GPU, you need different technique for that setting. I think we have never tested such small environment with only 1 GPU because it takes too much time to finish 90 epochs. Is this correct? @kfukuda
[Keisuke Fukuda, chainer] Yes, even the original ResNet author (Kaiming He) used 8 GPUs (if I remember correctly). The technique written on the paper is for extremely large minibatch size (>8K) and it does not make much sense to use the techniques (RMSProp warmup etc.) for such a small batchsize.
[Keisuke Fukuda, chainer] Before trying our techniques, I recommend to try Facebook’s result with 4K batchsize.
[Keisuke Fukuda, chainer] Techniques for large batch size does not necessarily good for small batchsize.
[Keisuke Fukuda, chainer] So, it depends on what you are trying to do @mingxiao.huang. If you want to reproduce our results, first you need to prepare >512 GPUs and try 32K batchsize. Trying to reproduce our results with small batchsize is nonsense. And, repeatedly, I recommend starting from Facebook’s result.
[Keisuke Fukuda, chainer] To achieve the SOTA with BS=128?
[Keisuke Fukuda, chainer] Or, reproduce our “extremely large minibatch” paper?