These are chat archives for beniz/deepdetect

30th
May 2018
Emmanuel Benazera
@beniz
May 30 2018 07:40
The travis cuda builds have been deactivated because they timeout and return failure flags. We've tried to lower the footprint, to no avail. TF / caffe2 builds take even longer so the builds have moved onto our sides for now. We're using gitlab to prepare the forthcoming PRs.
cchadowitz-pf
@cchadowitz-pf
May 30 2018 14:00
:+1: i'll get the exact errors but building with caffe+gpu is giving some issues with spdlog.
building with cmake .. -DUSE_CAFFE_CPU_ONLY=OFF (to ensure that caffe is built with cuda)
a number of errors similar to
/usr/include/spdlog/async_logger.h:64:308: error: converting to ‘const milliseconds {aka const std::chrono::duration<long int, std::ratio<1l, 1000l> >}’ from initializer list would use explicit constructor ‘constexpr std::chrono::duration<_Rep, _Period>::duration(const _Rep2&) [with _Rep2 = long int; <template-parameter-2-2> = void; _Rep = long int; _Period = std::ratio<1l, 1000l>]’
cchadowitz-pf
@cchadowitz-pf
May 30 2018 14:14
a snippet of the error+build output from make VERBOSE=1:
/usr/local/cuda/bin/nvcc -D_FORCE_INLINES -ccbin=g++ -Xcompiler -fPIC -DCAFFE_VERSION=1.0.0-rc3 -DNDEBUG -g -O2 -D_MWAITXINTRIN_H_INCLUDED -DUSE_OPENCV -DUSE_LEVELDB -DUSE_LMDB -I/usr/include/python2.7 -I/usr/lib/python2.7/dist-packages/numpy/core/include -I../../../protobuf/include -I/usr/local/include -I/usr/include/hdf5/serial -I/home/cchadowitz/include -I.build_release/src -I./src -I./include -I/usr/local/cuda/include -std=c++11 -gencode arch=compute_30,code=sm_30 -gencode arch=compute_35,code=sm_35 -gencode arch=compute_50,code=sm_50 -gencode arch=compute_52,code=sm_52 -gencode arch=compute_60,code=sm_60 -gencode arch=compute_61,code=sm_61 -gencode arch=compute_61,code=compute_61 -c src/caffe/layers/elu_layer.cu -o .build_release/cuda/src/caffe/layers/elu_layer.o 2> .build_release/cuda/src/caffe/layers/elu_layer.o.warnings.txt \
    || (cat .build_release/cuda/src/caffe/layers/elu_layer.o.warnings.txt; exit 1)
/usr/include/spdlog/spdlog.h:71:251: error: converting to ‘const milliseconds {aka const std::chrono::duration<long int, std::ratio<1l, 1000l> >}’ from initializer list would use explicit constructor ‘constexpr std::chrono::duration<_Rep, _Period>::duration(const _Rep2&) [with _Rep2 = long int; <template-parameter-2-2> = void; _Rep = long int; _Period = std::ratio<1l, 1000l>]’
 void set_async_mode(size_t queue_size, const async_overflow_policy overflow_policy = async_overflow_policy::block_retry, const std::function<void()>& worker_warmup_cb = nullptr, const std::chrono::milliseconds& flush_interval_ms = std::chrono::milliseconds::zero());
                                                                                                                                                                                                                                                           ^
cchadowitz-pf
@cchadowitz-pf
May 30 2018 14:20
I see in the travis builds it actually clones spdlog from github git clone https://github.com/gabime/spdlog.git - is there a known issue with the one in the ubuntu repos?
Emmanuel Benazera
@beniz
May 30 2018 14:39
hello, travis builds use Ubuntu 14.04 so we have to force spdlog via sources
on Ubuntu 16.04 it should work right out of the box
cchadowitz-pf
@cchadowitz-pf
May 30 2018 14:40
it was originally until about a day or two ago i believe
(i'm on Ubuntu 16.04)
but i'm now running into the issue above when trying to build caffe with cuda support
i'm on commit 8e1eb135650fafdfaa0ac9ae736d97f6946675a2 which i believe is the latest
Emmanuel Benazera
@beniz
May 30 2018 14:40
this looks like an nvcc nightmare, what version of Cuda are you on ?
cchadowitz-pf
@cchadowitz-pf
May 30 2018 14:41
$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2016 NVIDIA Corporation
Built on Wed_May__4_21:01:56_CDT_2016
Cuda compilation tools, release 8.0, V8.0.26
$ cat /usr/local/cuda/version.txt 
CUDA Version 8.0.27
CUDA Patch Version 8.0.27.1
i haven't changed that in a while so i'd be surprised if that was causing an issue, but it's entirely possible i suppose
cchadowitz-pf
@cchadowitz-pf
May 30 2018 14:53
trying it in a fresh docker container to make sure so i'll report back. if that works i guess i'll try updating my nvcc/cuda :-/
Emmanuel Benazera
@beniz
May 30 2018 14:58
are you saying that using -DUSE_CAFFE_CPU_ONLY=OFF still builds the CUDA version ?
oops, sorry missread
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:00
shouldn't it? i believe the only reason i left that in is because one time i hadn't cleaned my build folder (after using -DUSE_CAFFE_CPU_ONLY=ON) so it wasn't swapping to use CUDA in that case. but from an entirely clean build folder i can leave that out or use it set to OFF
hah no problem
Emmanuel Benazera
@beniz
May 30 2018 15:00
our main build machine cuda nvcc version is Cuda compilation tools, release 8.0, V8.0.61
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:00
that's the same as the latest nvidia/cuda docker image i believe, which so far appears to be working without a hitch. guess i'm due for an update!! sorry to bother you :)
Emmanuel Benazera
@beniz
May 30 2018 15:06
no pb, we could open an issue just for the sake of it and mark it for others...
I guess we all need to move to cuda 9 at some point
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:07
i'm not looking forward to that....i believe there may be some lack of support for >=cuda 9.2 and ubuntu 16.04 but i don't recall exactly
maybe that's specific to the gpu(s) we have here, i'm not sure.
Emmanuel Benazera
@beniz
May 30 2018 15:11
cuda 9.1 is fine I think, that's what we have on some of the production machines
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:11
right - 9.2 is where there may be some issues
does DD work ok with 9.1?
Emmanuel Benazera
@beniz
May 30 2018 15:12
yes, as far as I know
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:12
cool :+1:
Emmanuel Benazera
@beniz
May 30 2018 15:12
you'd need to update cudnn, drivers, etc...
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:12
right
Emmanuel Benazera
@beniz
May 30 2018 15:27
thanks for the quick issue :)
cchadowitz-pf
@cchadowitz-pf
May 30 2018 15:35
sure thing!