These are chat archives for beniz/deepdetect

19th
Dec 2016
ten2net
@ten2net
Dec 19 2016 10:31
when i train my model use dede-docker-gpu,[10:26:46] /opt/deepdetect/src/caffelib.cc:1703: iteration=0 ...src/caffelib.cc:1703: acc=0.717742...
[10:26:46] /opt/deepdetect/src/caffelib.cc:1730: exception while forward/backward pass through the network
Emmanuel Benazera
@beniz
Dec 19 2016 11:27
Usually a lack of memory. Look the server logs up.
Emmanuel Benazera
@beniz
Dec 19 2016 12:25
if not, submit a gist withe full server log.
Marc Reichman
@marcreichman
Dec 19 2016 14:24
@beniz: done beniz/deepdetect#227
Emmanuel Benazera
@beniz
Dec 19 2016 14:25
OK, I was to ask how it did go. Did you reproduce the original lock ?
Marc Reichman
@marcreichman
Dec 19 2016 14:26
no unfortunately, i more or less destroyed two dede's in two containers with two services each from 32 threads doing a series of 600K images, could not get them to lock
Emmanuel Benazera
@beniz
Dec 19 2016 14:26
good boy server ;)
Marc Reichman
@marcreichman
Dec 19 2016 14:26
i suspect having gdb attached is relevant.. i'm going to try another test without gdb to see if it matters, which i realize doesn't help much but would be nice to know we're not crazy over here :)
Emmanuel Benazera
@beniz
Dec 19 2016 14:27
ah yes, with gdb it slows down operations, which may change the outcome. The locking will reappear if you had it once, it means it is there.
Marc Reichman
@marcreichman
Dec 19 2016 14:28
had it twice on two diff boxes in that first period fwiw
Emmanuel Benazera
@beniz
Dec 19 2016 14:28
regarding #227 please include your steps building cppnetlib, etc... so that it can be reproduced easily.
rule of thumb being that copy/paste should allow to reproduce. This allows for quick verification, and quick action.
and make sure cppnetlib + boost headers are in your include path
Marc Reichman
@marcreichman
Dec 19 2016 14:32
they're in /usr/local/{lib,include} etc should i need special consideration for that in deepdetect? i would guess deepdetect cmake would set that up
that is supposed to be a gcc default path from what i can tell
Emmanuel Benazera
@beniz
Dec 19 2016 14:35
hmm, you better try cmake -DCMAKE_CXX_FLAGS='-I/usr/local/include' or something similar
errors like httpjsonapi.cc:217:48: error: 'response' in 'http_server {aka struct boost::network::http::server<APIHandler>}' does not name a type look like the headers are not found or have changed. If you can just make sure this is not the former.
Marc Reichman
@marcreichman
Dec 19 2016 14:38
on line 808 of the file i posted, there is reference to /usr/local/include/boost/network/....
i'm rebuilding with the explicit include path anyway but i think it is there
Emmanuel Benazera
@beniz
Dec 19 2016 14:39
yeah that's boost, not sure this is cppnetlib.
maybe it is, I will try in a moment as well
another question, have you tried 0.12 ?
Marc Reichman
@marcreichman
Dec 19 2016 14:43
haven't yet. was just basing on previous tickets where you suggested trying the latest cpp-netlib. distro came with 0.11.2, we tried master, 0.13, haven't explicitly tried 0.12 yet
Emmanuel Benazera
@beniz
Dec 19 2016 14:43
yep ok
Marc Reichman
@marcreichman
Dec 19 2016 15:00
explicitly adding the include path didn't change the outcome.
Emmanuel Benazera
@beniz
Dec 19 2016 15:19
Kinda hellish but I was able to reproduce. Looks like an API change. Try 0.12 if you can as this cannot be loooked at just now.
Marc Reichman
@marcreichman
Dec 19 2016 15:33
will try that and see if it makes a difference. for the record though, you recommend 0.12 over 0.11.2? i ask because the latter is already available via the distro, so it's fewer moving parts when we're building our containers
Emmanuel Benazera
@beniz
Dec 19 2016 15:34
yes distro is 11.2 but you may want to try 0.12, then run your tests again (with and without it). Then if you get the lock again, we'll need to find a way to investigate without gdb... since the lock doesn't happen with it...
Emmanuel Benazera
@beniz
Dec 19 2016 16:11
@marcreichman another suggestions, if you can keep track of the returned predict calls in your tests, so that any locking predict calls could be accounted for... it may help... Many are using dd over millions of calls, some have benchmarked the server beforehand, so either the locking is rare, either you are using in a new way. Still trying to find a lead.
Marc Reichman
@marcreichman
Dec 19 2016 16:56
will have to let you know on the hang. fwiw with some basic stuff i can't seem to get deepdetect building against cpp-netlib 0.12.0 either, i don't have time at the moment to dig in deeper
Emmanuel Benazera
@beniz
Dec 19 2016 16:57
I tried and was missing a buffer.hpp, I can't look it up more closely right now either.
Marc Reichman
@marcreichman
Dec 19 2016 16:58
i have said file in my /usr/include/boost/asio... but playing around with various -I in /usr/include and /usr/local/include made it find a file but then it blew up in other ways. for now i just know the 0.11.2 builds so i'm using it
Emmanuel Benazera
@beniz
Dec 19 2016 17:00
cppnetlib is annoying like that, one convenient way I found in the past is to take the pre-packaged sources from Ubuntu, and build from there. But I don't see 0.12 in the current list of future release, https://launchpad.net/ubuntu/+source/cpp-netlib
Tapabrata Ghosh
@sixsamuraisoldier
Dec 19 2016 18:36
Hey there
Beginner here.
I have a question
for the request in python
(for http POST /predict)
data array of strings no N/A array of data URI over which to make predictions, supports base64 for images
Could I use a file name here instead of an URL?
Also will the DD server work in Java/Android
I mean
will the request work in android/java
W. L.
@wardloockx
Dec 19 2016 18:59
Hey Guys, I'm trying to setup the example. But when creating the service with the PUT command I get
{"status":{"code":400,"msg":"BadRequest","dd_code":1006,"dd_msg":"Service Bad Request Error"}}
This is with the pure example code
anyone that can help out ?
Marc Reichman
@marcreichman
Dec 19 2016 19:15
@wardloockx What OS?
@sixsamuraisoldier should work fine.
Tapabrata Ghosh
@sixsamuraisoldier
Dec 19 2016 19:44
@marcreichman awesome, thanks!
@marcreichman but we have to import the dede server even in the client for python though, right?
Marc Reichman
@marcreichman
Dec 19 2016 19:46
@sixsamuraisoldier i'm sorry, i don't know anything about dede + python. my experience is just running the service and using rest
Tapabrata Ghosh
@sixsamuraisoldier
Dec 19 2016 19:46
ah nvm, my bad dd_client seems to just be a REST library for calling DD through urllib2. Convenient!
@marcreichman no worries, thanks for your help :)
cchadowitz-pf
@cchadowitz-pf
Dec 19 2016 19:48
@sixsamuraisoldier There's a small example on using the python client on https://deepdetect.com/tutorials/imagenet-classifier/
from dd_client import DD
dd = DD('localhost')
dd.set_return_format(dd.RETURN_PYTHON)
description = 'image classification service'
mllib = 'caffe'
model = {'templates':'../templates/caffe/','repository':'../../models/imgnet'}
parameters_input = {'connector':'image'}
parameters_mllib = {'template':'googlenet','nclasses':1000}
parameters_output = {}
dd.put_service('imageserv',model,description,mllib,
               parameters_input,parameters_mllib,parameters_output)
Tapabrata Ghosh
@sixsamuraisoldier
Dec 19 2016 19:56
@cchadowitz-pf Thanks, my confusion was in what the dd_client was doing, I didn't realize it was just a REST library.
cchadowitz-pf
@cchadowitz-pf
Dec 19 2016 20:09
No problem :)