Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Jan 27 19:42
    mergify[bot] unlabeled #1499
  • Jan 27 19:42

    mergify[bot] on master

    feat: log model parameters and … (compare)

  • Jan 27 19:42
    mergify[bot] closed #1499
  • Jan 27 19:42
    mergify[bot] labeled #1499
  • Jan 27 17:25
    Bycob synchronize #1499
  • Jan 27 15:49
    Bycob commented #1499
  • Jan 27 15:48
    mergify[bot] review_requested #1499
  • Jan 27 15:48
    mergify[bot] review_requested #1499
  • Jan 27 15:48
    mergify[bot] review_requested #1499
  • Jan 27 15:47
    Bycob labeled #1499
  • Jan 27 15:47
    Bycob labeled #1499
  • Jan 27 15:47
    Bycob opened #1499
  • Jan 25 13:39
    mergify[bot] unlabeled #1497
  • Jan 25 13:39

    mergify[bot] on master

    feat(torch): data augmentation … (compare)

  • Jan 25 13:39
    mergify[bot] closed #1497
  • Jan 25 13:39
    mergify[bot] labeled #1497
  • Jan 25 11:33
    royale unlabeled #1497
  • Jan 25 11:32
    royale synchronize #1497
  • Jan 25 11:28
    mergify[bot] unlabeled #1498
  • Jan 25 11:27

    mergify[bot] on master

    fix(torch): avoid race conditio… (compare)

dgtlmoon
@dgtlmoon
Emmanuel Benazera
@beniz
we'll see, it's most likely from misusing the faiss API, try "index_type":"IVF20,SQ8" alone... but FAISS has endless options/parameters, so you may want to buidl a small benchmark.
dgtlmoon
@dgtlmoon
yeah i agree, alright, i'll try that.. thanks again
should be some good FAISS indexing examples in other projects I can look at too
Emmanuel Benazera
@beniz
there's an autotune to the indexes, you may want to look at this as well, maybe you can optimize the index a posteriori
https://github.com/facebookresearch/faiss/blob/main/demos/demo_sift1M.cpp good luck... maybe there's an example in python...
dgtlmoon
@dgtlmoon
"IVF20,SQ8" caused all sorts of errors (a few pages above here, segfaults, weird errors etc)
Emmanuel Benazera
@beniz
it might be the ondisk, and the building schedule of the index
https://github.com/facebookresearch/faiss/wiki/Faiss-indexes#flat-indexes if faiss defaults to a flat index, your search time is related to the size of the index (which is a stupid thing on their part IMO). Annoy would do better right out of the box.
dgtlmoon
@dgtlmoon
ok, and autotune is a feature/attribute of different index_types right ? https://github.com/facebookresearch/faiss/wiki/The-index-factory
yes! sounds exactly like it
yeah why build an amazing library, which is off by defaults :D
dgtlmoon
@dgtlmoon
hmm how to get the number of vectors in my index.faiss
dgtlmoon
@dgtlmoon
indexing with "index_type": "IVF262144_HNSW32,PQ64", "train_samples": 100000, "nprobe": 64 seems to work.. dont know what i was doing wrong, i think leaving off the "ondisk: true" helped
indexing 120k/images, lets see how it goes in about 24 hours, I'll be able todo the simsearch again and compare the search/query time
dgtlmoon
@dgtlmoon
dgtlmoon
@dgtlmoon
weird, index.faiss is 8.1mb, even after 2000 images, after building index with "index":false, "build_index":true
Emmanuel Benazera
@beniz
if your train_samples is 100000, you very certainly want to build the index after 100k images are in...
dgtlmoon
@dgtlmoon
yeah, srry, that was silly
i'll let it run the full 120k and see how we go
Emmanuel Benazera
@beniz
you may want to set 100 and test it with 100 first :)
dgtlmoon
@dgtlmoon
train_samples 100, at 350 images, I dumped the index to disk, and I see the same 8~Mb
at 6000 images... it is still 8Mb
weird
i'll try a different index type, "IVF262144_HNSW32,PQ64" maybe too brutal
dgtlmoon
@dgtlmoon
IVF in combination with HNSW uses HNSW to do the cluster assignment. You will need between 30 * 65536 and 256 * 65536 vectors for training. yeah ok, so at the time of dumping the index to disk, maybe theres not enough vectors
dgtlmoon
@dgtlmoon

@beniz going back to what's in the deepdetect docs.... It's segfaulting...
I index with..

 'output': {'index': True, "ondisk": True, "index_type": "IVF20,SQ8", "train_samples": 100,  "nprobe": 64  }

Index looks good.. 6000 images

180K    model/index.faiss
31M     model/index_mmap.faiss
curl -X PUT "http://localhost:8080/services/test" -d '{
       "mllib":"caffe",
       "description":"similarity search service",
       "type":"unsupervised",
       "parameters":{
         "input":{
           "connector":"image",
           "height": 224,
           "width": 224
         },
         "mllib":{
           "nclasses":20
         }
       },
       "model":{
          "templates":"../templates/caffe/",
          "repository":"/var/www/xxx/web/files-tshirt/trainer/simsearch/model/",
          "weight": "model_iter_13500.caffemodel"
       }
     }'
$ curl -X POST "http://localhost:8080/predict" -d '{
>        "service":"test",
>        "parameters":{
>          "input":{ "height": 224, "width": 224  },
>          "output":{ "search_nn": 10, "search": true  },
>          "mllib":{ "extract_layer":"pool5/7x7_s1"  }
>        },
>        "data":["https://www.google.com/images/branding/googlelogo/1x/googlelogo_color_272x92dp.png"]  }'
curl: (52) Empty reply from server

[2021-11-14 21:29:29.883] [test] [info] Using pre-trained weights from /var/www/xxx/web/files-tshirt/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-14 21:29:30.211] [torchlib] [info] Attempting to upgrade batch norm layers using deprecated params: /var/www/xxx/web/files-tshirt/trainer/simsearch/model/model_iter_13500.caffemodel
[2021-11-14 21:29:30.211] [torchlib] [info] Successfully upgraded batch norm layers using deprecated params.
[2021-11-14 21:29:30.315] [test] [info] Net total flops=3858534272 / total params=26063936
[2021-11-14 21:29:30.315] [test] [info] detected network type is classification
[2021-11-14 21:29:30.315] [api] [info] HTTP/1.1 "PUT /services/test" <n/a> 201 551ms
open existing index db
[2021-11-14 21:30:08.347] [torchlib] [info] Opened lmdb /var/www/xxx/web/files-tshirt/trainer/simsearch/model//names.bin
bash: line 1:     7 Segmentation fault      (core dumped) ./dede -host 0.0.0.0
i'm able to reproduce the segfault every time
jolibrain_cpu

GIT REF: heads/v0.19.0:1673a99ecc922e01dd7cc8845098291ef46a8902
COMPILE_FLAGS: USE_CAFFE2=OFF USE_TF=OFF USE_NCNN=ON USE_TORCH=OFF USE_HDF5=ON USE_CAFFE=ON USE_TENSORRT=OFF USE_TENSORRT_OSS=OFF USE_DLIB=OFF USE_CUDA_CV=OFF USE_SIMSEARCH=ON USE_ANNOY=OFF USE_FAISS=ON USE_COMMAND_LINE=ON USE_JSON_API=ON USE_HTTP_SERVER=OFF
DEPS_VERSION: OPENCV_VERSION=4.2.0 CUDA_VERSION= CUDNN_VERSION= TENSORRT_VERSION=
so i'm unable to use the IVF indexes without segfault... I notice that when I send the query, the CPU is working for a few seconds... and then... segfault
I'm able to reproduce
so maybe an issue between DD and the FAISS IVF lookup?
dgtlmoon
@dgtlmoon
hmm maybe i can use GDB and debug build to get a core dump and then stacktrace
Emmanuel Benazera
@beniz
you can submit an issue along with a script to replicate on public data or any image set, that'd speed resolution up
dgtlmoon
@dgtlmoon
dgtlmoon
@dgtlmoon
yeah maybe try different version of faiss, hmm
dgtlmoon
@dgtlmoon
oh man simsearch GPU training is fast x)
question, should "train_samples": 10000, be the TOTAL size of all of your images you expect to train in the set? or just a nice localised number for where it will compare against?
say i have 150k images, maybe 20,000 might be a good choice?
dgtlmoon
@dgtlmoon
i guess depends on how much time VS accuracy you want
dgtlmoon
@dgtlmoon
ahhh yeahhhhhhhhh 0.070s query time for simsearch x) yesss
dgtlmoon
@dgtlmoon
I would <3 if https://www.deepdetect.com/server/docs/api/ was on github so I can add some improvements
2 replies
dgtlmoon
@dgtlmoon
max(ninvertedlist/50,2) what does invertedlist mean in this case?
tasibalint
@tasibalint
image.png
Anyone an idea what this could mean?
sry i t possible that class 1 train images are 36 and the other are 44 and i use a batch size of 5, and the test images are 6 for each classe i am gonna fix that first
Emmanuel Benazera
@beniz
@tasibalint this message means that the mean_valuefile is wrong somehow, not sure what you did exactly, mind sharing the API calls / steps you are using ?
Emmanuel Benazera
@beniz
or are your image b&w ?
tasibalint
@tasibalint
I have done the cats_dogs tutorial, and now out of desperation i started the cats_dogs training with my images and the training is running, soo apperantly the .cafemodel i was using wasn't compatible or something.
I was not using the model from:
"init": "https://deepdetect.com/models/init/desktop/images/classification/ilsvrc_googlenet.tar.gz",
But from :
https://github.com/BVLC/caffe/tree/master/models/bvlc_googlenet