beniz/deepdetect

Deep Learning API and Server

dgtlmoon

@dgtlmoon

https://github.com/facebookresearch/faiss/blob/main/demos/demo_auto_tune.py

dgtlmoon

@dgtlmoon

indexing with "index_type": "IVF262144_HNSW32,PQ64", "train_samples": 100000, "nprobe": 64 seems to work.. dont know what i was doing wrong, i think leaving off the "ondisk: true" helped

indexing 120k/images, lets see how it goes in about 24 hours, I'll be able todo the simsearch again and compare the search/query time

dgtlmoon

@dgtlmoon

looks like some nice benchmarking scripts https://github.com/facebookresearch/faiss/blob/main/benchs/README.md

dgtlmoon

@dgtlmoon

weird, index.faiss is 8.1mb, even after 2000 images, after building index with "index":false, "build_index":true

Emmanuel Benazera

@beniz

if your train_samples is 100000, you very certainly want to build the index after 100k images are in...

dgtlmoon

@dgtlmoon

yeah, srry, that was silly

i'll let it run the full 120k and see how we go

Emmanuel Benazera

@beniz

you may want to set 100 and test it with 100 first :)

dgtlmoon

@dgtlmoon

train_samples 100, at 350 images, I dumped the index to disk, and I see the same 8~Mb

at 6000 images... it is still 8Mb

weird

i'll try a different index type, "IVF262144_HNSW32,PQ64" maybe too brutal

dgtlmoon

@dgtlmoon

IVF in combination with HNSW uses HNSW to do the cluster assignment. You will need between 30 * 65536 and 256 * 65536 vectors for training.

yeah ok, so at the time of dumping the index to disk, maybe theres not enough vectors

dgtlmoon

@dgtlmoon

@beniz going back to what's in the deepdetect docs.... It's segfaulting...
I index with..

 'output': {'index': True, "ondisk": True, "index_type": "IVF20,SQ8", "train_samples": 100,  "nprobe": 64  }

Index looks good.. 6000 images

180K    model/index.faiss
31M     model/index_mmap.faiss

curl -X PUT "http://localhost:8080/services/test" -d '{
       "mllib":"caffe",
       "description":"similarity search service",
       "type":"unsupervised",
       "parameters":{
         "input":{
           "connector":"image",
           "height": 224,
           "width": 224
         },
         "mllib":{
           "nclasses":20
         }
       },
       "model":{
          "templates":"../templates/caffe/",
          "repository":"/var/www/xxx/web/files-tshirt/trainer/simsearch/model/",
          "weight": "model_iter_13500.caffemodel"
       }
     }'

$ curl -X POST "http://localhost:8080/predict" -d '{
>        "service":"test",
>        "parameters":{
>          "input":{ "height": 224, "width": 224  },
>          "output":{ "search_nn": 10, "search": true  },
>          "mllib":{ "extract_layer":"pool5/7x7_s1"  }
>        },
>        "data":["https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"]  }'
curl: (52) Empty reply from server


[2021-11-14 21:29:29.883] [test] [info] Using pre-trained weights from /var/www/xxx/web/files-tshirt/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-14 21:29:30.211] [torchlib] [info] Attempting to upgrade batch norm layers using deprecated params: /var/www/xxx/web/files-tshirt/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-14 21:29:30.211] [torchlib] [info] Successfully upgraded batch norm layers using deprecated params.
[2021-11-14 21:29:30.315] [test] [info] Net total flops=3858534272 / total params=26063936
[2021-11-14 21:29:30.315] [test] [info] detected network type is classification
[2021-11-14 21:29:30.315] [api] [info] HTTP/1.1 "PUT /services/test" <n/a> 201 551ms
open existing index db
[2021-11-14 21:30:08.347] [torchlib] [info] Opened lmdb /var/www/xxx/web/files-tshirt/trainer/simsearch/model//names.bin
bash: line 1:     7 Segmentation fault      (core dumped) ./dede -host 0.0.0.0

i'm able to reproduce the segfault every time

jolibrain_cpu


GIT REF: heads/v0.19.0:1673a99ecc922e01dd7cc8845098291ef46a8902
COMPILE_FLAGS: USE_CAFFE2=OFF USE_TF=OFF USE_NCNN=ON USE_TORCH=OFF USE_HDF5=ON USE_CAFFE=ON USE_TENSORRT=OFF USE_TENSORRT_OSS=OFF USE_DLIB=OFF USE_CUDA_CV=OFF USE_SIMSEARCH=ON USE_ANNOY=OFF USE_FAISS=ON USE_COMMAND_LINE=ON USE_JSON_API=ON USE_HTTP_SERVER=OFF
DEPS_VERSION: OPENCV_VERSION=4.2.0 CUDA_VERSION= CUDNN_VERSION= TENSORRT_VERSION=

so i'm unable to use the IVF indexes without segfault... I notice that when I send the query, the CPU is working for a few seconds... and then... segfault

I'm able to reproduce

so maybe an issue between DD and the FAISS IVF lookup?

dgtlmoon

@dgtlmoon

hmm maybe i can use GDB and debug build to get a core dump and then stacktrace

Emmanuel Benazera

@beniz

you can submit an issue along with a script to replicate on public data or any image set, that'd speed resolution up

dgtlmoon

@dgtlmoon

jolibrain/deepdetect#1368 :)

dgtlmoon

@dgtlmoon

yeah maybe try different version of faiss, hmm

dgtlmoon

@dgtlmoon

oh man simsearch GPU training is fast x)

question, should "train_samples": 10000, be the TOTAL size of all of your images you expect to train in the set? or just a nice localised number for where it will compare against?

say i have 150k images, maybe 20,000 might be a good choice?

dgtlmoon

@dgtlmoon

i guess depends on how much time VS accuracy you want

dgtlmoon

@dgtlmoon

ahhh yeahhhhhhhhh 0.070s query time for simsearch x) yesss

dgtlmoon

@dgtlmoon

I would <3 if https://www.deepdetect.com/server/docs/api/ was on github so I can add some improvements

2 replies

dgtlmoon

@dgtlmoon

max(ninvertedlist/50,2) what does invertedlist mean in this case?

tasibalint

@tasibalint

Anyone an idea what this could mean?

sry i t possible that class 1 train images are 36 and the other are 44 and i use a batch size of 5, and the test images are 6 for each classe i am gonna fix that first

Emmanuel Benazera

@beniz

@tasibalint this message means that the mean_valuefile is wrong somehow, not sure what you did exactly, mind sharing the API calls / steps you are using ?

Emmanuel Benazera

@beniz

or are your image b&w ?

tasibalint

@tasibalint

I have done the cats_dogs tutorial, and now out of desperation i started the cats_dogs training with my images and the training is running, soo apperantly the .cafemodel i was using wasn't compatible or something.
I was not using the model from:
"init": "https://deepdetect.com/models/init/desktop/images/classification/ilsvrc_googlenet.tar.gz",
But from :
https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet

same name same size but different D: anyways i am training now :D

Emmanuel Benazera

@beniz

yeah, you need to use ours :)

tasibalint

@tasibalint

Hey people how do u train bbox detection model and where do u get the model from?

Emmanuel Benazera

@beniz

@tasibalint maybe too broad of a question... you can train via API or platform. What are you trying to achieve ?

tasibalint

@tasibalint

I want to train vai API, I have created with lableImg xml files with bounding boxes for each picture. But i dont understand how I tell the api to use those xml files

Service create

             "description":"generic image detection  service",
             "model":{
                 "repository":"/images/models/packages_detc",    
                 "templates":"../templates/caffe/",
                 "weight": "SE-ResNet-50.caffemodel"

            },
            "mllib":"caffe",
            "type":"supervised",
            "parameters":{
                "input":{
                    "connector":"image",
                    "width":224,
                    "height":224,
                    "db": true,
                    "bbox": true
                },
                "mllib":{
                  "template":"resnet_50",
                  "nclasses":3,
                  "finetuning":true
                }
            },
        }

        "service":"packages_detc",
        "async":true,
        "parameters":{
            "input":{
                "connector":"image",
                "test_split":0.1,
                "shuffle":true,
                "height":224,
                "width":224,
                "db":true,
                "rgb":true,
            },
            "mllib":{
                "gpu":false,
                "mirror":true,
                "net":{
                    "batch_size":3,
                    "test_batch_size":3
                },
                "solver":{
                    "test_interval":500,
                    "iterations":1000,
                    "base_lr":0.001,
                },
                "noise":{"all_effects":true, "prob":0.001},
                "distort":{"all_effects":true, "prob":0.01},
                "bbox": true
            },
            "output":{
                 "measure":["acc","mcll","f1"],
            }
        },
        "data":["/images/train/"]
    }

Getting the error " auto batch size set to zero:" but i dont get it where it is set to zero

On the platform object detection guide i see this

is there a toll to convert xml files into what ever this txt is for a format, without doing it manually?

Emmanuel Benazera

@beniz

this is not a template for object detection, you'd need to use one of the ssd templates and get a dataset in proper format, see https://www.deepdetect.com/platform/docs/object-detection/ it has the format description

tasibalint

@tasibalint

thank you i am still unsure what the difference between mllib, model and templates are

Any ideas in how i can see what's in the:
"templates":"../templates/caffe/",
folder or if there are any other templates then caffe. is this where the docker is installed and if yes, is there a generic path where the docker is isntalled at?

tasibalint

@tasibalint

where do i define the classes for the ssd_300 model? i have <label> <xmin> <ymin> <xmax> <ymax> where label is a number. but how do i define that the number is a class, for classification models there is the corresp.txt, but for detection models?

Where communities thrive

People

Repo info

Activity