beniz/deepdetect

Deep Learning API and Server

dgtlmoon

@dgtlmoon

yeah i agree, alright, i'll try that.. thanks again

should be some good FAISS indexing examples in other projects I can look at too

Emmanuel Benazera

@beniz

there's an autotune to the indexes, you may want to look at this as well, maybe you can optimize the index a posteriori

https://github.com/facebookresearch/faiss/blob/main/demos/demo_sift1M.cpp good luck... maybe there's an example in python...

dgtlmoon

@dgtlmoon

"IVF20,SQ8" caused all sorts of errors (a few pages above here, segfaults, weird errors etc)

Emmanuel Benazera

@beniz

it might be the ondisk, and the building schedule of the index

https://github.com/facebookresearch/faiss/wiki/Faiss-indexes#flat-indexes if faiss defaults to a flat index, your search time is related to the size of the index (which is a stupid thing on their part IMO). Annoy would do better right out of the box.

dgtlmoon

@dgtlmoon

ok, and autotune is a feature/attribute of different index_types right ? https://github.com/facebookresearch/faiss/wiki/The-index-factory

yes! sounds exactly like it

yeah why build an amazing library, which is off by defaults :D

dgtlmoon

@dgtlmoon

hmm how to get the number of vectors in my index.faiss

dgtlmoon

@dgtlmoon

https://github.com/facebookresearch/faiss/blob/main/demos/demo_auto_tune.py

dgtlmoon

@dgtlmoon

indexing with "index_type": "IVF262144_HNSW32,PQ64", "train_samples": 100000, "nprobe": 64 seems to work.. dont know what i was doing wrong, i think leaving off the "ondisk: true" helped

indexing 120k/images, lets see how it goes in about 24 hours, I'll be able todo the simsearch again and compare the search/query time

dgtlmoon

@dgtlmoon

looks like some nice benchmarking scripts https://github.com/facebookresearch/faiss/blob/main/benchs/README.md

dgtlmoon

@dgtlmoon

weird, index.faiss is 8.1mb, even after 2000 images, after building index with "index":false, "build_index":true

Emmanuel Benazera

@beniz

if your train_samples is 100000, you very certainly want to build the index after 100k images are in...

dgtlmoon

@dgtlmoon

yeah, srry, that was silly

i'll let it run the full 120k and see how we go

Emmanuel Benazera

@beniz

you may want to set 100 and test it with 100 first :)

dgtlmoon

@dgtlmoon

train_samples 100, at 350 images, I dumped the index to disk, and I see the same 8~Mb

at 6000 images... it is still 8Mb

weird

i'll try a different index type, "IVF262144_HNSW32,PQ64" maybe too brutal

dgtlmoon

@dgtlmoon

IVF in combination with HNSW uses HNSW to do the cluster assignment. You will need between 30 * 65536 and 256 * 65536 vectors for training.

yeah ok, so at the time of dumping the index to disk, maybe theres not enough vectors

dgtlmoon

@dgtlmoon

@beniz going back to what's in the deepdetect docs.... It's segfaulting...
I index with..

 'output': {'index': True, "ondisk": True, "index_type": "IVF20,SQ8", "train_samples": 100,  "nprobe": 64  }

Index looks good.. 6000 images

180K    model/index.faiss
31M     model/index_mmap.faiss

curl -X PUT "http://localhost:8080/services/test" -d '{
       "mllib":"caffe",
       "description":"similarity search service",
       "type":"unsupervised",
       "parameters":{
         "input":{
           "connector":"image",
           "height": 224,
           "width": 224
         },
         "mllib":{
           "nclasses":20
         }
       },
       "model":{
          "templates":"../templates/caffe/",
          "repository":"/var/www/xxx/web/files-tshirt/trainer/simsearch/model/",
          "weight": "model_iter_13500.caffemodel"
       }
     }'

$ curl -X POST "http://localhost:8080/predict" -d '{
>        "service":"test",
>        "parameters":{
>          "input":{ "height": 224, "width": 224  },
>          "output":{ "search_nn": 10, "search": true  },
>          "mllib":{ "extract_layer":"pool5/7x7_s1"  }
>        },
>        "data":["https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"]  }'
curl: (52) Empty reply from server


[2021-11-14 21:29:29.883] [test] [info] Using pre-trained weights from /var/www/xxx/web/files-tshirt/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-14 21:29:30.211] [torchlib] [info] Attempting to upgrade batch norm layers using deprecated params: /var/www/xxx/web/files-tshirt/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-14 21:29:30.211] [torchlib] [info] Successfully upgraded batch norm layers using deprecated params.
[2021-11-14 21:29:30.315] [test] [info] Net total flops=3858534272 / total params=26063936
[2021-11-14 21:29:30.315] [test] [info] detected network type is classification
[2021-11-14 21:29:30.315] [api] [info] HTTP/1.1 "PUT /services/test" <n/a> 201 551ms
open existing index db
[2021-11-14 21:30:08.347] [torchlib] [info] Opened lmdb /var/www/xxx/web/files-tshirt/trainer/simsearch/model//names.bin
bash: line 1:     7 Segmentation fault      (core dumped) ./dede -host 0.0.0.0

i'm able to reproduce the segfault every time

jolibrain_cpu


GIT REF: heads/v0.19.0:1673a99ecc922e01dd7cc8845098291ef46a8902
COMPILE_FLAGS: USE_CAFFE2=OFF USE_TF=OFF USE_NCNN=ON USE_TORCH=OFF USE_HDF5=ON USE_CAFFE=ON USE_TENSORRT=OFF USE_TENSORRT_OSS=OFF USE_DLIB=OFF USE_CUDA_CV=OFF USE_SIMSEARCH=ON USE_ANNOY=OFF USE_FAISS=ON USE_COMMAND_LINE=ON USE_JSON_API=ON USE_HTTP_SERVER=OFF
DEPS_VERSION: OPENCV_VERSION=4.2.0 CUDA_VERSION= CUDNN_VERSION= TENSORRT_VERSION=

so i'm unable to use the IVF indexes without segfault... I notice that when I send the query, the CPU is working for a few seconds... and then... segfault

I'm able to reproduce

so maybe an issue between DD and the FAISS IVF lookup?

dgtlmoon

@dgtlmoon

hmm maybe i can use GDB and debug build to get a core dump and then stacktrace

Emmanuel Benazera

@beniz

you can submit an issue along with a script to replicate on public data or any image set, that'd speed resolution up

dgtlmoon

@dgtlmoon

jolibrain/deepdetect#1368 :)

dgtlmoon

@dgtlmoon

yeah maybe try different version of faiss, hmm

dgtlmoon

@dgtlmoon

oh man simsearch GPU training is fast x)

question, should "train_samples": 10000, be the TOTAL size of all of your images you expect to train in the set? or just a nice localised number for where it will compare against?

say i have 150k images, maybe 20,000 might be a good choice?

dgtlmoon

@dgtlmoon

i guess depends on how much time VS accuracy you want

dgtlmoon

@dgtlmoon

ahhh yeahhhhhhhhh 0.070s query time for simsearch x) yesss

dgtlmoon

@dgtlmoon

I would <3 if https://www.deepdetect.com/server/docs/api/ was on github so I can add some improvements

2 replies

dgtlmoon

@dgtlmoon

max(ninvertedlist/50,2) what does invertedlist mean in this case?

tasibalint

@tasibalint

Anyone an idea what this could mean?

sry i t possible that class 1 train images are 36 and the other are 44 and i use a batch size of 5, and the test images are 6 for each classe i am gonna fix that first

Emmanuel Benazera

@beniz

@tasibalint this message means that the mean_valuefile is wrong somehow, not sure what you did exactly, mind sharing the API calls / steps you are using ?

Emmanuel Benazera

@beniz

or are your image b&w ?

tasibalint

@tasibalint

I have done the cats_dogs tutorial, and now out of desperation i started the cats_dogs training with my images and the training is running, soo apperantly the .cafemodel i was using wasn't compatible or something.
I was not using the model from:
"init": "https://deepdetect.com/models/init/desktop/images/classification/ilsvrc_googlenet.tar.gz",
But from :
https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet

same name same size but different D: anyways i am training now :D

Emmanuel Benazera

@beniz

yeah, you need to use ours :)

Where communities thrive

People

Repo info

Activity