Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Oct 30 22:29
    fantes commented #895
  • Oct 30 22:29
    fantes commented #895
  • Oct 30 10:31

    BynaryCobweb on master

    docs: Remove trailing whitespac… (compare)

  • Oct 30 10:31
    BynaryCobweb closed #923
  • Oct 30 10:29
    BynaryCobweb labeled #923
  • Oct 30 10:29
    BynaryCobweb labeled #923
  • Oct 30 10:29
    BynaryCobweb labeled #923
  • Oct 30 10:29
    BynaryCobweb opened #923
  • Oct 30 09:57
    BynaryCobweb commented #909
  • Oct 30 09:54
    BynaryCobweb synchronize #909
  • Oct 29 16:02
    beniz edited #909
  • Oct 29 15:13
    BynaryCobweb review_requested #909
  • Oct 29 15:13
    BynaryCobweb review_requested #909
  • Oct 29 15:13
    BynaryCobweb review_requested #909
  • Oct 29 15:13
    BynaryCobweb labeled #909
  • Oct 29 15:13
    BynaryCobweb labeled #909
  • Oct 29 15:13
    BynaryCobweb labeled #909
  • Oct 29 15:05
    BynaryCobweb synchronize #909
  • Oct 28 09:44
    beniz commented #895
  • Oct 28 08:52
    YaYaB commented #895
Emmanuel Benazera
@beniz
libmapbox-variant-dev and rapidjdon-dev are not in 16.04 but are header-based, so easy to setup
Yes we'll discuss backward compatibility warnings internally, and I believe they'll appear first in commit messages, then automatically in the changelog
At the moment 18.04 is the safe bet, as some more libraries are gone in 20.04 and we'll change some internals accordingly
cchadowitz-pf
@cchadowitz-pf
:+1: one reason we've stuck with 16.04 is that overall system memory usage went up when we tried 18.04. I think I mentioned that here a while back. Since we're sometimes in resource-limited scenarios, it was better for us to stay with 16.04 in that situation. Do you happen to have any general sense of memory usage overall between different ubuntu versions on your end? Wondering if you've seen anything similar
Emmanuel Benazera
@beniz
Never observed this. Memory in inference with what backend ? GPU ?
cchadowitz-pf
@cchadowitz-pf
I believe CPU, but it's been a while since I've checked. I'll try to do some more rigorous comparisons and let you know
Emmanuel Benazera
@beniz
For resource limited scenarios with CPU, the ncnn backend might be useful.
dgtlmoon
@dgtlmoon
@cchadowitz-pf dont forget to change the engine type on CPU backends when you've trained on a GPU
"parameters": { "mllib": { "nclasses": 7, "gpu": false, "engine":"CUDNN_MIN_MEMORY" }, (for example with a resnet50 classifier)
@beniz i've got fairly nice accuracy with my classifier, https://www.deepdetect.com/server/docs/imagenet-classifier/ question is - this will give me the best classification for the image, but what if i want to know multiple things about the image? I train on colours of tshirts, but i want to know if the tshirt is longsleeve or shortsleeve... so should I add a black-longsleeve black-shortsleeve category, or can i train on black green shortsleeve longsleeve ? it will only attempt to pick the best types out of 1.0 right? 0.999999 black but then shortsleeve wont look so good? i hope you understand my question :)
dgtlmoon
@dgtlmoon
the docs page menu titles says Image tagger but really it is Image classification... tagging would mean it has more than one valid tag/classification
Emmanuel Benazera
@beniz
Yes you are looking for multi label classification. It's supported though maybe not part of the documentation (yet)
dgtlmoon
@dgtlmoon
@beniz ahuh, wonderful :) is that some kind of option to enable or is that the default mode for classification? (doesnt seem so?)
https://gitter.im/beniz/deepdetect/archives/2016/10/09 sort of related perhaps, yeah caffe+lmdb use multiple dbs
Emmanuel Benazera
@beniz
hi @dgtlmoon this is not properly documented at the moment but you'll find instructions here: jolibrain/deepdetect#381
This is actually a good time for us to document things, so please continue to add your feedback. We may not always answer but we do notice, and have a sprint on doc coming up soon. Ping me if you need help.
YaYaB
@YaYaB
Hey,
Congrats for the last release of https://github.com/jolibrain/deepdetect/releases/tag/v0.10.1. I see that things are evolving fast and I have some questions:
  • Do you plan to support new data connectors for libtorch?
  • Will it be possible to define specific preprocessing for a libtorch model (use specific libraries for data augmentation, etc.)?
Emmanuel Benazera
@beniz
hey @YaYaB :
  • what data connectors are you interested in ? at the moment image classification, time series and NLP (Bert/GPT) are supported.
  • data augmentation will be available for libtorch
    In general, you can ask/tell the team here about features that you believe would be useful in your context, based on relevance, usefulness, feasibility, working bandwidth, etc. they can be scheduled.
    Regarding libtorch, we'd like to transfer most architectures to it, but pytorch still suffers from a very exploded ecosystem, and most models are not production ready. If you look around for object detection models, it's hard to find implementations that for instance transfer to tensorrt easily, if not to C++... For this reason, we're moving first the models and tasks that most strongly benefit libtorch: recurrent models (e.g. LSTMs, OCR) and complex / new architectures (e.g. nbeats, generative models).
YaYaB
@YaYaB
@beniz
SVM connector might be interesting for me as well. By the way, the connector graphs in https://github.com/jolibrain/deepdetect might need some updates.
Good to know for the features. I understand your point about the different architectures. However if you keep the latest version of libtorch up to date we could infer using a model trained directly with pytorch?
Emmanuel Benazera
@beniz
  • SVM connector for libtorch: at the moment if interested, you may want to try contributing it, probably connecting to the new native libtorch interface in DD. You can ping @fantes if you need info about this
  • What do you mean by connector graphs exactly ?
  • Running pytorch models requires converting them to torchscript or tracing them. Support for this is lacking in most pytorch contributed ecosystem libraries, so it may or may not work, you can ping @BynaryCobweb on these matters
YaYaB
@YaYaB
@beniz thanks for the information. I mean the table "Input data support per library (current):" does not indicate that LibTorch supports Time-Series nor Text characters.
Emmanuel Benazera
@beniz
good call, it's still mixed within the input connectors API doc, the doc is to be refactored soon.
dgtlmoon
@dgtlmoon
@beniz yes no problem for me, you know I love to improve documentation :)
dgtlmoon
@dgtlmoon
do you ever "make up" or create images to fill a training/validating dataset?
I guess there would be situations where you simply had to
Emmanuel Benazera
@beniz
sometimes we build / use synthetic data generators yes
dgtlmoon
@dgtlmoon
I'm training on about 5000 images for different cat/class, but sometimes i find that the background really influences the image categorisation... i guess it's best to have a chain setup with a detector first right?
or is there some deepdetect API training tricks?
Emmanuel Benazera
@beniz
Hello, set noise_prob and distort_prob to 0.01 or 0 005 as data augmentation ? It's hard to know what influence categorisation.
dgtlmoon
@dgtlmoon
yeah, sorry for the vague question
the object (a shirt) is about 80% of the image, but sometimes it's 50% of the image with a lot of background 'noise', but the actual shirt is clear/clean
I remember something about a random zoom OUT and IN on training?
Emmanuel Benazera
@beniz
Use the distort_prob as well with zoom_in and zoom_out activated. For object detection models zoomed augmentation is already on.
dgtlmoon
@dgtlmoon
ahh yes, zoom_in will run according to distort_prob, sorry, i thought it was for all images... yeah so i'll set it to 0.01 or so
dgtlmoon
@dgtlmoon
thanks :)
dgtlmoon
@dgtlmoon
https://www.deepdetect.com/server/docs/api/ <-- is noise available for prediction (i'de like to add contrast at prediction time)? the docs arent clear if that distort section works ... as mllib doesnt belong in prediction, right? bit lost in the docs there... tried this call (adding a little contrast at prediction time really helps my accuracy)

curl -X POST 'http://dd:8080/predict' -d '{
  "service": "classifier",
  "parameters": {
    "input": {
       "noise": {
        "contrast": true
       }
    },
    "output": {
      "confidence_threshold": 0.01,
      "best": 5
    },
    "mllib": {
      "gpu": true,
       "noise": {
        "contrast": true
       }
    }
  },
  "data": [
    "'$1'"
  ]
}'
dgtlmoon
@dgtlmoon
histogram equalisation looked good on some images but really damaged other images
Emmanuel Benazera
@beniz
Noise is only at training time
dgtlmoon
@dgtlmoon
no problems, will do it in another process
dgtlmoon
@dgtlmoon
ha ha ha, my project is mainly concerned about identifying objects by color... so maybe disabling the distort options todo with color is a nice idea :D :D
      "distort": {
        "prob": 0.02,
        "random ordering": false,
        "HUE": false
      },
      "noise": {
        "gauss_blur": true,
        "prob": 0.02,
        "decolorize": false,
        "hist_eq": false,
        "inverse": false,
        "posterize": false,
        "clahe": false,
        "convert_to_hsv": false,
        "convert_to_lab": false
      },
se_resnet_50 works well
        "Error": {
            "code": 500,
            "msg": "InternalError",
            "dd_code": 1007,
            "dd_msg": "basic_string::substr: __pos (which is 140) > this->size() (which is 0)"
        }
dgtlmoon
@dgtlmoon
wow, using an old working recipe, but with new data (training jpgs)
unsure
jolibrain/deepdetect_gpu docker version and GIT REF: master:ec2f5e27470371d995e640d1f6c2722d08415051 COMPILE_FLAGS: USE_CAFFE2=OFF USE_TF=OFF USE_NCNN=OFF USE_TORCH=OFF USE_HDF5=ON USE_CAFFE=ON USE_TENSORRT=OFF USE_TENSORRT_OSS=OFF USE_DLIB=OFF USE_CUDA_CV=OFF USE_SIMSEARCH=ON USE_ANNOY=OFF USE_FAISS=ON USE_COMMAND_LINE=ON USE_JSON_API=ON USE_HTTP_SERVER=ON DEPS_VERSION: OPENCV_VERSION=3.2.0 CUDA_VERSION_STRING=10.2 CUDNN_VERSION=
same error from jolibrain/deepdetect_cpu
is there some filename length limit?
dgtlmoon
@dgtlmoon
longest path+filename i have is 117chars
dgtlmoon
@dgtlmoon
I think adding -strip fixed it convert $subname -resize 500x500\> -strip classify/ready-to-train/$bname/$bname.$i.$ext