These are chat archives for beniz/deepdetect

29th
Feb 2016
Dawid Wolski
@merito
Feb 29 2016 13:59

Hello again. I try to tun training from images but it fails. First it looks like it is working properly, the images are read but it hangs on
INFO - Network initialization done.
Then when I try to check job status using
curl -X GET "http://localhost:8080/train?service=myimages&job=1"
answer is
{"status":{"code":200,"msg":"OK"},"head":{"method":"/train","job":1,"status":"error"},"body":{}}
and server instantaneously shows
INFO - Solver scaffolding done.
ERROR - service myimages training status call failed

ERROR - {"code":400,"msg":"BadRequest","dd_code":1006,"dd_msg":"Service Bad Request Error"}

Emmanuel Benazera
@beniz
Feb 29 2016 14:36
hey @merito
you're trying to what ? finetune ?
Dawid Wolski
@merito
Feb 29 2016 14:38
I create a service, it is fine, next I start to train the net
Emmanuel Benazera
@beniz
Feb 29 2016 14:39
you'll need to show the output of the server
Dawid Wolski
@merito
Feb 29 2016 14:39
and when it stops on Network initialization done message (I've tested it during the night) I can't get trainig status and server crashed
Emmanuel Benazera
@beniz
Feb 29 2016 14:41
server crash means something really really bad with Caffe, in general... Best might be to discuss and full server output. Maybe best in private chat, you decide.
Dawid Wolski
@merito
Feb 29 2016 14:41
I'll paste a link here in a minutes
with full server output
Emmanuel Benazera
@beniz
Feb 29 2016 14:42
good, put the calls with it, removing any private stuff.
I've compiled dede with CUDA support, but there is no matter if I run training with gpu true or false, it behaves the same
Emmanuel Benazera
@beniz
Feb 29 2016 14:53
ok, looking
@merito resume is set to true, is that intentional ?
Dawid Wolski
@merito
Feb 29 2016 14:58
oh, no, it shouldn't be there
Emmanuel Benazera
@beniz
Feb 29 2016 15:00
here you go, this is most likely the culprit
which we should make easier to debug I reckon
Dawid Wolski
@merito
Feb 29 2016 15:00
sorry for disturbing you with this
Emmanuel Benazera
@beniz
Feb 29 2016 15:01
well, happened to me many times :)
that resume thing...
gonna fix it now with a more obvious error message in the server logs...
Emmanuel Benazera
@beniz
Feb 29 2016 15:09
done
Dawid Wolski
@merito
Feb 29 2016 15:11
thank you. Now I'm trying to run it on GPU. Is there a need to configure something to use GPU when CUDA was discovered by cmake?
Emmanuel Benazera
@beniz
Feb 29 2016 15:13
there shouldn't be, but CUDA is its own beast...
nvidia-smi is your friend
Dawid Wolski
@merito
Feb 29 2016 15:13
yeah, I know
right now on my console
Ozéias Sant'ana
@ozeias
Feb 29 2016 22:43
hey @beniz
i'm getting several errors like this: "ERROR - service objects prediction call failed"
and after while the service dies
do you have any idea why?