in your setups, what do you use primarily to pass image data to DD? base64 encoding, local file paths, remote file URLs, or something else?
caching the URL content locally and serving with local webserver can work fine
if you see other types of convenient setup, happy to hear about them
we've used local filepaths when we can mount local dirs in our docker environments, but we often have DD running on a separate system (and in docker) from where the image files are, so in those cases we host a local webserver and pass the urls.
I was just thinking of ways to minimize disk IO and was trying to think if there were any other ways to pass by memory besides base64 but can't really think of anything. not familiar with how you can use protobufs as http payloads though, sounds interesting!
I was just thinking of ways to minimize disk IO and was trying to think if there were any other ways to pass by memory besides base64 but can't really think of anything. not familiar with how you can use protobufs as http payloads though, sounds interesting!
Protobuf+http is the easiest alternative yet. We are changing the http server to oatpp, and this will open the way to protobuf+http. OATPP is actually already built into DD but we moved it back to secondary because of a very strange issue with tbb intel library that is used by OpenCV. Very annoying rare bug that is slowing our migration down and thus a series of improvements. That's the game I guess...
We have traced it back to tbb, a detached thread id just vanishes, we're trying various things now. I believe it's proven an OpenCV build with another thread lib does not lead to the crash. Very annoying issue. It seems tbb is bound to get in the C++ standard as an implementation, so we need to find either a fix, either a workaround. OATPP is very well made and its dev are very attentive to every detail. Great lib from great people!
I gone this link,
https://www.deepdetect.com/platform/docs/training-from-csv-data/
But still not cleared to me
if you are using the DeepDetect server, the doc for CSV is https://www.deepdetect.com/server/docs/csv-training/
@beniz In the above link it has train.csv file has been trained but want my own dataset from my csv file, so for that i have to train my own csv file
change your CPU :)
we have cached versions of the torch build
you can tarball the pre-built torch as needed, same for caffe
Haha I'll ask for more ressources :p
On my side it seems to be stuck quite a long time on faiss with the following errors
On my side it seems to be stuck quite a long time on faiss with the following errors
ptxas /tmp/tmpxft_00006a63_00000000-12_PQScanMultiPassNoPrecomputed.compute_50.ptx, line 111; warning : ld
...
...
ptxas /tmp/tmpxft_0000753a_00000000-5_PQScanMultiPassPrecomputed.compute_75.ptx, line 11479; warning : ld
they get detected by cmake, and you can also force them by hand for caffe with CUDA_ARCH, not sure about pytorch builds, I can ask those who deal with it.
there'll be an internal card for passing native gpu arch
hey @beniz - continuing to experiment with NCNN, I was trying to convert the word_detect_v2, crop action, multiword_ocr chain from Caffe to NCNN and ran into an internal error:
[2021-01-15 22:07:29.135] [ocr-d21cfe6c-2e8c-40c7-94df-b4740bcfb44a-0] [info] number of calls=3
[2021-01-15 22:07:29.135] [ocr-d21cfe6c-2e8c-40c7-94df-b4740bcfb44a-0] [info] [0] / executing predict on service word_detect_v2_ncnn
[2021-01-15 22:07:35.575] [ocr-d21cfe6c-2e8c-40c7-94df-b4740bcfb44a-0] [info] [1] / executing action crop
[2021-01-15 22:07:35.576] [api] [error] 10.10.10.32 "PUT /chain/ocr-d21cfe6c-2e8c-40c7-94df-b4740bcfb44a-0" 500 6441msand the returned error was:
{
"status": {
"code": 500,
"dd_code": 1007,
"dd_msg": "in get<T>()",
"msg": "InternalError"
}
}Any idea what's going on?
@cchadowitz-pf OK thanks for the report, got it, see PR #1137. There should be a 0.12.1 release next wed anyways as we fixed a few things.
However, there's a remaining issue with NCNN since it only supports batch size of 1, and thus on chains, downstream NCNN models can only process the first sample.
I'll add larger batch size support for NCNN, but their doc shows how to do this by simply using omp to parallelize a for loop, meaning it's parallel on the CPU, not aggregated into batches at GPU level.
However, there's a remaining issue with NCNN since it only supports batch size of 1, and thus on chains, downstream NCNN models can only process the first sample.
I'll add larger batch size support for NCNN, but their doc shows how to do this by simply using omp to parallelize a for loop, meaning it's parallel on the CPU, not aggregated into batches at GPU level.
Nevertheless, since NCNN is mostly supposed to be used on CPU, this should not harm too much. Maybe I'll force the number of threads to the number of local CPU cores.
I wasn't aware that NCNN was even utilizing the GPU in DeepDetect since it relies on vulkan I thought?
but nvidia gpus via vulkan I believe
