These are chat archives for beniz/deepdetect

25th
Oct 2017
rperdon
@rperdon
Oct 25 2017 12:27
I'm working on ensuring the matix I'm reading and processing in extract layers is correct
rperdon
@rperdon
Oct 25 2017 13:43
I've been playing with the input of extract_layer_nsfw.py and have identified up to what point I can insert the openCV code to supplant the PIL and process it correctly. I think the key lies in ensuring that the caffe.io.load_image is still called on raw data after a resize function has been called.
rperdon
@rperdon
Oct 25 2017 13:57
I have successfully re-written extract_layer_nsfw to open an image using OpenCV (with BGR to RGB flip) and the outputted flattened files are identical.
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:02
So the image loaded with OpenCV with the BGR->RGB swap is identical to the image loaded with PIL?
rperdon
@rperdon
Oct 25 2017 14:06
That is correct.
I can send you my rewritten extract_layer_nsfw.py which inputs using OpenCV
Emmanuel Benazera
@beniz
Oct 25 2017 14:09
so what's the current leading hypothesis ? the discrepancies would be due to RGB vs BGR ?
rperdon
@rperdon
Oct 25 2017 14:09
Ran filecompare on the outputs
they matched
I believe its just RGB vs BGR
Emmanuel Benazera
@beniz
Oct 25 2017 14:11
OK, if this is the issue, this can be set as an API option very easily
rperdon
@rperdon
Oct 25 2017 14:11
PIL also includes alpha channel information, but my classify.py OpenCV modification did not cause any divergences in classifications
Use this example.py with the classification example code here
This example will do an OpenCV load
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:13
isn't your line 168 in extract_layer_nsfw.py still overwriting the image_data with the PIL data?
image_data = scipy.misc.imresize(image_data, (256, 256), 'bilinear') image_data = Image.fromarray(image_data)
or is it converting the scipy array to PIL format?
rperdon
@rperdon
Oct 25 2017 14:16
I was thinking of converting back into something after I flipped it from RGB to BGR
I think its in np ndarray format
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:18
I'd be curious to see if using opencv from file-read to passing it to the caffe transformer (no Image.fromarray() and no converting to raw data then converting the raw data back to an np ndarray) is the same as using PIL throughout with the raw data conversion. In theory there shouldn't be a need to convert the opencv np ndarray to raw data and back (though I understand that you were following the existing code).
I guess what I'm saying is passing the result of this
image_data = scipy.misc.imresize(image_data, (256, 256), 'bilinear')
directly to caffe.io.load_image() or even better, directly to caffe_transformer.preprocess(), if either is possible
rperdon
@rperdon
Oct 25 2017 14:19
I was following existing code, trying to ensure that the only key differences would be OpenCV load vs PIL, but I noticed that the OpenCV format is np array already
I will try playing with it further
image_data = scipy.misc.imresize(image_data, (256, 256), 'bilinear')
I think when I tried to pass that directly, it gave an error about the np array
which is why I converted it to PIL data
but up until that point the information was from OpenCV
image_data = scipy.misc.toimage(image_data)
rperdon
@rperdon
Oct 25 2017 14:25
I used this to convert to raw image data for opencv and it worked out like the pil conversion
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:30
so the caffe_transformer.preprocess() method expects a (H' x W' x K) ndarray from the documentation: https://github.com/BVLC/caffe/blob/master/python/caffe/io.py#L122
so i wonder if that would let you skip the raw image data step entirely...
rperdon
@rperdon
Oct 25 2017 14:35
I'll look into it
caffe.io.load_image
I noticed something about the matrix loaded at this point; my earlier code I sent in an email to you DID send the information directly to the caffe_transformer.preprocess()
the result was those very large numbers in the flattened output
I imagine if this step was skipped then the PIL flattend output should come out the same as well.
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:46
the step of converting to raw image data?
the caffe.io.load_image call implies RGB, by the way: https://github.com/BVLC/caffe/blob/master/python/caffe/io.py#L279
rperdon
@rperdon
Oct 25 2017 14:54
I skipped the caffe.io.load_image and fed the caffe_transformer.preprocess() the same data as inputted from PIL and OpenCV while still doing the resize functions for both
the results outputted were identical
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:57
awesome! it looks like that's different than when the raw data is passed in. So I wonder if the model was trained on the raw data input but DD is using the standard pixel values like the numpy ndarray values
rperdon
@rperdon
Oct 25 2017 14:58
It is possible with regards to the yahoo nsfw model
I know for the models I work with from digits that it works with numpy ndarray values when trained
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 14:59
so then this wouldn't really resolve the issue for your models
rperdon
@rperdon
Oct 25 2017 14:59
I think the bgr vs rgb may be the key issue for my models
provided dd uses the same numpy ndarray values for input
Emmanuel Benazera
@beniz
Oct 25 2017 15:07
Then it's easy to test with DD: switch the two channels for an image before you dump it to disk, then pass it to DD, and see whether you get the expected result
if the test shows this is the source of the problem, we'll add an RGB/BGR flag to the input API
rperdon
@rperdon
Oct 25 2017 15:17
I guess I have some conversions to do :)
rperdon
@rperdon
Oct 25 2017 16:02
I have a question on DeepDetect on its image load: with Digits based classification, there is use of image squash and mean pixel subtraction. The models are trained using this method as well. Is this something that is taken into account when loading images in DeepDetect?
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 16:03
mean pixel subtraction is taken care of either with a mean.binaryproto file or specifying an RGB value as the global mean pixel value
image squash i don't think is applied, but that's definitely something @beniz should answer
rperdon
@rperdon
Oct 25 2017 16:06
I was looking at that already and see it is optional for the mean pixel subtraction
The models do have a mean.binaryprotofile in the folder
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 16:08
then DD should be picking those up to use when loading the model. I do think that the file needs to be named exactly mean.binaryproto though
see #313 for details about my findings with that :)
rperdon
@rperdon
Oct 25 2017 16:14
Confirmed it is named correctly.
The other thing that has bugged me has been the image resize; when I change it to 256/256 it doesn't work. I have tried altering the deploy.prototxt to 256
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:14
depends on what the model was trained on. for the yahoo model it was expecting i believe 256x256 but internally center-cropped it to 224x224
rperdon
@rperdon
Oct 25 2017 17:34
My models were trained on 256x256, I thought the yahoo model also used 256 as well.
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:35
i don't recall, you could be right actually. what resize value does work for you, if 256 doesn't?
rperdon
@rperdon
Oct 25 2017 17:36
It was only working with 224 values
which I found odd
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:37
is it possible your training prototxt was applying a center crop? that's what i've found in general, that images are resized to 256x256 then center-cropped to 224x224 when training
rperdon
@rperdon
Oct 25 2017 17:39
When selecting the training, you are shown what resize is applied. I selected the squash method, not center crop
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:41
oh, through DIGITS? i haven't played with that myself yet
rperdon
@rperdon
Oct 25 2017 17:42
With the 256 values, when the deploy and call for the classifcation occurs, a strange thing happens , the output is not a 0, 1 classification despite being only a binary classifier. I get odd categories like 5,6, 4, 5 and occasionally 0, 1
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:43
in deepdetect?
rperdon
@rperdon
Oct 25 2017 17:43
from deep detect
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:44
that's bizarre. are you able to share your deploy.prototxt?
With it set to 224, 224, I was getting back 0,1 or 1, 0 depending on the classification
{"service":"animeAN","parameters":{"input":{"width":256,"height":256},"output":{
"best":2},"mllib":{"gpu":false}},"data":["/images/invert_test07.png"]}
The output of the call to the deepdetect server
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:49
what call are you using to create the deepdetect service?
rperdon
@rperdon
Oct 25 2017 17:49
curl -X PUT "http://127.0.0.1:9999/services/animeAN" -d "{\"mllib\":\"caffe\",\"description\":\"image anime or not\",\"type\":\"supervised\",\"parameters\":{\"input\":{\"connector\":\"image\"},\"mllib\":{\"nclasses\":2}},\"model\":{\"repository\":\"/models/AnimeAN/\"}}"
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:51
not sure if it makes a difference, but i actually set the width/height in the PUT service creation call, not the predict call
"parameters":{ "input":{ "connector":"image", "width":224, "height":224} for example
rperdon
@rperdon
Oct 25 2017 17:52
I can try that there. That would mean I would not need to supply it to the call to the server?
service
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:53
right
for predicting, i typically use something like:
{
        "service":"'${THISMODEL}'",
        "parameters":{
            "output":{
                "best":1
                 }
        },
               "data":["/opt/dede/cat.jpg"]
    }
rperdon
@rperdon
Oct 25 2017 17:54
curl -X PUT "http://127.0.0.1:9999/services/animeAN" -d "{\"mllib\":\"caffe\",\"description\":\"image anime or not\",\"type\":\"supervised\",\"parameters\":{\"input\":{\"connector\":\"image\", \"width":256, \"height\":256},\"mllib\":{\"nclasses\":2}},\"model\":{\"repository\":\"/models/AnimeAN/\"}}"
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:54
looks like you may be missing a \
rperdon
@rperdon
Oct 25 2017 17:54
I'll try it out, report in a few mins
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 17:54
:+1:
rperdon
@rperdon
Oct 25 2017 17:54
on width, I see it
rperdon
@rperdon
Oct 25 2017 18:04
So I removed the width and height call. Same oddity
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 18:07
may have to ask @beniz about that :-/ not sure how or why it'd be giving something besides 0 or 1 for a binary classifier.
rperdon
@rperdon
Oct 25 2017 18:07
The "best" option is still returning an odd value; it should return a 0 or a 1 for its category (it is a binary classifier), but the category returned is >1 ie I've seen 4,5,6 depending on the image
I lowered the number to deepdetect's default of 227
at that value; the classifications are returning 0/1 again
rperdon
@rperdon
Oct 25 2017 18:14
Also sadly, inverting the images did not correct the results produced from deep detect.
I'm back to sqaure one trying to determine the root cause
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 18:14
by inverting you mean swapping their channels from rgb -> bgr?
rperdon
@rperdon
Oct 25 2017 18:14
yes
I inverted 8 test images which were being classified wrong
and re ran them after being inverted, the results indicate another factor at work other than bgr/rgb
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 18:26
hmm
rperdon
@rperdon
Oct 25 2017 18:28
So possible that the image loaded is using another method like crop
instead of the expected squash. I will train a model based on some other image functions and try to infer what deepdetect is using.
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 18:52
so do you think the rgb/bgr issue is a factor (but not the only factor), or is it not affecting it after all?
rperdon
@rperdon
Oct 25 2017 18:52
the results are affected by rgb/bgr
but it is not the only factor
With the digits classifier, flipping to bgr caused a large divergence in results
cchadowitz-pf
@cchadowitz-pf
Oct 25 2017 18:54
mm
rperdon
@rperdon
Oct 25 2017 19:02
I should have 3 50k image alexnet model each trained on crop, fill, halfcrop/fill trained in the next 30 mins
rperdon
@rperdon
Oct 25 2017 20:16
We will wait on Beniz to provide input on how deepdetect works for the image load.