These are chat archives for beniz/deepdetect

5th
Jun 2018
cchadowitz-pf
@cchadowitz-pf
Jun 05 2018 18:32
seems like tests are broken again for caffe? i have the following list of failed tests using the latest commit in master:
[  FAILED  ] 23 tests, listed below:
[  FAILED  ] caffeapi.service_train
[  FAILED  ] caffeapi.service_train_async_status_delete
[  FAILED  ] caffeapi.service_train_async_final_status
[  FAILED  ] caffeapi.service_train_async_and_predict
[  FAILED  ] caffeapi.service_predict
[  FAILED  ] caffeapi.service_train_csv
[  FAILED  ] caffeapi.service_train_csv_in_memory
[  FAILED  ] caffeapi.service_train_csv_resnet
[  FAILED  ] caffeapi.service_train_svm
[  FAILED  ] caffeapi.service_train_svm_resnet
[  FAILED  ] caffeapi.service_train_images
[  FAILED  ] caffeapi.service_train_images_imagedatalayer_1label
[  FAILED  ] caffeapi.service_train_images_imagedatalayer_multilabel
[  FAILED  ] caffeapi.service_train_images_imagedatalayer_multilabel_softprob
[  FAILED  ] caffeapi.service_train_images_convnet
[  FAILED  ] caffeapi.service_train_images_resnet
[  FAILED  ] caffeapi.service_train_images_seg
[  FAILED  ] caffeapi.service_train_txt
[  FAILED  ] caffeapi.service_train_txt_sparse
[  FAILED  ] caffeapi.service_train_txt_sparse_lr
[  FAILED  ] caffeapi.service_train_txt_char
[  FAILED  ] caffeapi.service_train_txt_char_resnet
[  FAILED  ] caffeapi.service_train_csv_mt_regression
they seem to all be due to {"code":500,"msg":"InternalError","dd_code":1007,"dd_msg":"src/caffe/common.cpp:170 / Check failed (custom): (error) == (cudaSuccess)"}
and i see that before it fails, it's showing Using GPU 2
seems like it's now defaulting to GPU 2 for the tests for some reason (i have 2 gpus total, so i'm assuming they'd be GPU 0 and GPU 1, not GPU 2)
Emmanuel Benazera
@beniz
Jun 05 2018 20:35
Yes the gpu 2 got stuck there after a merge, we saw that earlier. Really we should have it on the command line