These are chat archives for beniz/deepdetect

26th
Oct 2017
Emmanuel Benazera
@beniz
Oct 26 2017 08:20
@cchadowitz-pf @rperdon regarding squash, that if I understand correctly is resizing, this is done by DD's input connector using a cubic filter, see https://github.com/beniz/deepdetect/blob/master/src/imginputfileconn.h#L82
Emmanuel Benazera
@beniz
Oct 26 2017 08:32
@rperdon I don't think it's numerically possible to get a category > 1 on a binary classifier. If you share your model + all prototxt and mean.binaryproto, I'll look at it but my hypothesis is that something is wrong elsewhere.
Emmanuel Benazera
@beniz
Oct 26 2017 08:38
The main source of difference must be in the inputs, as found by @cchadowitz-pf when comparing PIL vs OpenCV and skimage. The second source might be NVidia Caffe vs Berkely Caffe, and I can't comment on that one. From what you guys have done, could you answer the following question, as it would be useful: when feeding DD and the Digit Python script with the exact same inputs (i.e. exact same per-pixel value), are you getting the same output, or not ? If you don't know yet, in order to print the read image values inside DD, go to the link to the image readers in the input connector as shared above, print the values and see whether they match what the Python script says.
Now, if this is too complicated for some reason, please provide a package that contains a pre-trained model + all required files + scripts so that I can reproduce and I'll look into it. Bear in mind that any badly trained / overfit model will behave strangely around even tiny modifications. So it'd be best to pick a publicly available digit model for instance and that would be known to behave well in general.
rperdon
@rperdon
Oct 26 2017 13:04
I will look into it. I looked at the sizing and I believe 227 was the correct value so no worries on the category bug.
When I check the inpur values of PIL and OpenCV, I believe I printed out the per-pixel values and noted the alpha channel difference in PIL, but I will confirm that.
I did a full output per-pixel load as well for both:
image = cv2.imread(path);
print "OpenCV load"


width, height, shape = image.shape

for i in range(width):
   for j in range(height):
       k = image[i,j]
       print k
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
sub = PIL.Image.open(path)

pixels = sub.load()
width, height = sub.size

print "PIL load"
all_pixels = []
for x in range(width):
    for y in range(height):
        cpixel = pixels[x, y]
        all_pixels.append(cpixel)
        print cpixel
It seems on the surface that with my digit python script, that openCV and PIL are reading the same input image (-the BGR line)
In terms of the Digits input vs output, openCV and PIL are producing the same output
rperdon
@rperdon
Oct 26 2017 13:54
I will look into getting the input data per-pixel data of DD to compare
Emmanuel Benazera
@beniz
Oct 26 2017 13:55
question is: if you pass the BGR image to Digit, what do you get ?
In order to test DD with RGB images, you can try to apply the following patch:
diff --git a/src/imginputfileconn.h b/src/imginputfileconn.h
index 98a47dc..91da209 100644
--- a/src/imginputfileconn.h
+++ b/src/imginputfileconn.h
@@ -101,6 +101,7 @@ namespace dd
     int read_file(const std::string &fname)
     {
       cv::Mat img = cv::imread(fname,_bw ? CV_LOAD_IMAGE_GRAYSCALE : CV_LOAD_IMAGE_COLOR);
+      cv::cvtColor(img,img,CV_BGR2RGB);
       if (img.empty())
        {
          LOG(ERROR) << "empty image";
@@ -191,6 +192,7 @@ namespace dd
       for (std::pair<std::string,int> &p: lfiles)
        {
          cv::Mat img = cv::imread(p.first,_bw ? CV_LOAD_IMAGE_GRAYSCALE : CV_LOAD_IMAGE_COLOR);
+         cv::cvtColor(img,img,CV_BGR2RGB);
          _imgs_size.push_back(std::pair<int,int>(img.rows,img.cols));
          cv::Mat rimg;
          cv::resize(img,rimg,size,0,0,CV_INTER_CUBIC);
rperdon
@rperdon
Oct 26 2017 14:04
Feeding BGR images to Digits caused major deviation in the classification.
Emmanuel Benazera
@beniz
Oct 26 2017 14:04
OK good
rperdon
@rperdon
Oct 26 2017 14:05
my dd is loaded in ubuntu right now
Emmanuel Benazera
@beniz
Oct 26 2017 14:05
so maybe try the patch above (patch -p1 < thepatch.txt), recompile DD and then use the API to pass an image to your Digit model
rperdon
@rperdon
Oct 26 2017 14:07
Save code as thepatch.txt, load dd into bash console then "patch" it inside?
Emmanuel Benazera
@beniz
Oct 26 2017 14:07
cd deepdetect then apply the patch command above
rperdon
@rperdon
Oct 26 2017 14:07
ah k, give me a few mins to do that
rperdon
@rperdon
Oct 26 2017 14:22
-it bash is not letting me into console of deepdetect
entrypoint; forgot that
rperdon
@rperdon
Oct 26 2017 14:35
formatting of the patch is biting me
rperdon
@rperdon
Oct 26 2017 14:41
dd@c65d4997aed5:/opt/deepdetect$ patch -p1 < thepatch.txt
patching file src/imginputfileconn.h
Hunk #1 FAILED at 101.
Hunk #2 FAILED at 191.
2 out of 2 hunks FAILED -- saving rejects to file src/imginputfileconn.h.rej
dd@c65d4997aed5:/opt/deepdetect$
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 14:45
are you using the latest version? you could edit imginputfileconn.h directly instead if the patch is being a nuisance
Emmanuel Benazera
@beniz
Oct 26 2017 14:46
yeah, might be easier to just make the changes yourself then
rperdon
@rperdon
Oct 26 2017 14:46
thx
I will edit it directly then
I downloaded dd mid october; any updates since?
rperdon
@rperdon
Oct 26 2017 15:07
I edited the file and added the 2 lines
I just need to commit the change
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:07
i think you just need to rebuild DD and restart the server, unless you're in a container of some sort and need to commit the change to the container image
rperdon
@rperdon
Oct 26 2017 15:12
how do I rebuild dd after the h file change?
I just finished committing the change, but dd exited with code 0
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:15
committing the change in git? it depends on how you're running DD. if you cloned the git repo and built it yourself locally, all you need to do now is go through the same steps for building (excluding re-cloning the repo, as that would wipe out this change).
rperdon
@rperdon
Oct 26 2017 15:28
I git cloned the repo and just ran it
I have a docker-compose.yml file that calls on the beniz/deepdetect_cpu
so I've not built it per say
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:29
ahh ok you're using a docker container.
rperdon
@rperdon
Oct 26 2017 15:29
I did a docker commit to save the changed file
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:30
you will then also need to get into the docker bash session and rebuild DD inside the container (and commit that as well)
rperdon
@rperdon
Oct 26 2017 15:32
I'm in the container bash now
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:34
ok, you may be able to just cd deepdetect/build && make then...
maybe before make you may need do do a cmake .. again, not sure what your setup is though
rperdon
@rperdon
Oct 26 2017 15:39
I called make again...
warnings so far, waiting on it to build
rperdon
@rperdon
Oct 26 2017 15:51
dd is exiting with code 0 still
I ran make and it said it completed the build
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:51
exiting with code 0 means no errors, i would assume
rperdon
@rperdon
Oct 26 2017 15:54
its ignoring my docker-compse.yml file now?
normally it should load
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 15:55
the code change shouldn't have affected anything regarding docker. unless somehow something changed with the docker config
rperdon
@rperdon
Oct 26 2017 15:57
I'll take lunch and rethink this in the pm
I added 2 lines, commited changes, went back in, ran make, build successful, then reran again but now the server is exiting rather than starting up
I may just enter into bash manually map volumes and run the dede command to "load" models
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 16:00
yup i think that's a good idea, then you can know for sure what's going on
rperdon
@rperdon
Oct 26 2017 17:24
I need to play around calling ./dede correctly with the parameters I've set
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:31
if you do docker images, does it show that beniz/deepdetect_cpu was updated when you committed that change?
i forget if it commits it to a new image name
rperdon
@rperdon
Oct 26 2017 17:32
I used the same image name
cpcmecadmin@mlhost:/media/shared/DeepDetect$ sudo docker ps -l
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
cb0281ac5886 beniz/deepdetect_cpu "/bin/bash" 12 minutes ago Exited (0) 5 seconds ago ecstatic_benz
cpcmecadmin@mlhost:/media/shared/DeepDetect$
I find it amusing that when troubleshooting that my copy/paste is adding sad faces
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:33
:) try surrounding code blocks with three backticks (`)
code
single backticks for inline code
i'd try running dede manually inside the container
rperdon
@rperdon
Oct 26 2017 17:33
so from what I get of docker, it is "working"
ie I was able to save the changes to the .h file
Beniz had me add in 2 lines of code based on his snippit
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:34
yup i saw that
rperdon
@rperdon
Oct 26 2017 17:34
I saved the file back into the src folder
then I went into build
and ran make
which outputted a successful build
my only weird oddity is why it is NOT following the yml code and loading up deep detect listen server
I'm reading into docker on how to initiate the network bridge portion of the code
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:36
:+1:
rperdon
@rperdon
Oct 26 2017 17:51
when in bash, I am able to start dede: on stopping (ctrl-c) I get the following error

ERROR - 17:51:46 - std::future_error: No associated state

ERROR - 17:51:46 - Bad file descriptor

cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:52
but while it's running it's all good? loads models and all that?
rperdon
@rperdon
Oct 26 2017 17:53
I can get dede started; but I lose access to it afterwards
I am unable to call curl commands after running ./dede
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:54
you should be able to make curl calls from a second terminal session while dede is running
rperdon
@rperdon
Oct 26 2017 17:54
that's where it is not working

dd@f8c79a56b982:/opt/deepdetect/build/main$ ./dede
DeepDetect [ commit 978401f3d1f23a327d0ebfef24cb0a0d7c543c6e ]

INFO - 17:56:12 - Running DeepDetect HTTP server on localhost:8080

cpcmecadmin@mlhost:/media/shared/DeepDetect$ curl -X PUT "http://127.0.0.1:8080/services/animeAN" -d "{\"mllib\":\"caffe\",\"description\":\"image anime or not\",\"type\":\"supervised\",\"parameters\":{\"input\":{\"connector\":\"image\", \"width\":227, \"height\":227},\"mllib\":{\"nclasses\":2}},\"model\":{\"repository\":\"/models/AnimeAN/\"}}"
curl: (7) Failed to connect to 127.0.0.1 port 8080: Connection refused
docker learning curve for me
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 17:58
sounds like when you started the container you didn't expose port 8080 :)
docker run -p 8080:8080 beniz/deepdetect_cpu or similar
rperdon
@rperdon
Oct 26 2017 18:03
Recv failure: Connection reset by peer
looking up error now
rperdon
@rperdon
Oct 26 2017 18:53
-p command opens up the port, I can ping 127.0.0.1, I can redis-cli but I am refused
rperdon
@rperdon
Oct 26 2017 19:11
I have hit a wall for today
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:16
so when i've worked with DD with docker, i haven't used a docker-compose.yml to start it up. i've used a command like the one above to start up the container with the given port mapping (and optionally, some directory mapping) and then use curl calls to set up services with appropriate models and run a couple classification/prediction calls
using the command above should start up the docker image beniz/deepdetect_cpu mapping the local port 8080 to the container port 8080
so then if you do something like curl GET http://localhost:8080/info it should give you some info on the deepdetect server running in the container
rperdon
@rperdon
Oct 26 2017 19:22
curl GET http://localhost:8080/infocurl: (6) Could not resolve host: GET
curl: (56) Recv failure: Connection reset by peer

sudo docker run -v /media/shared/DeepDetect/images:/images -v /media/shared/DeepDetect/models:/models -p 8080:8080 -it --entrypoint /bin/bash beniz/deepdetect_cpu
dd@607620fb0b4a:/opt/deepdetect/build/main$ ./dede
DeepDetect [ commit 978401f3d1f23a327d0ebfef24cb0a0d7c543c6e ]

INFO - 19:23:07 - Running DeepDetect HTTP server on localhost:8080

cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:23
sorry, forgot a -X in my example. try this:
curl -X GET "http://localhost:8080/info"
rperdon
@rperdon
Oct 26 2017 19:23
2 different terminal windows 1 says alive, the other says I can't reach
curl: (56) Recv failure: Connection reset by peer
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:24
what's the output of sudo docker ps?
rperdon
@rperdon
Oct 26 2017 19:24
sudo docker ps
[sudo] password for cpcmecadmin:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
607620fb0b4a beniz/deepdetect_cpu "/bin/bash" 2 minutes ago Up 2 minutes 0.0.0.0:8080->8080/tcp xenodochial_mcnulty
c6b9244c4b3b beniz/deepdetect_cpu "/bin/bash" 4 hours ago Restarting (0) 2 seconds ago deepdetect
cpcmecadmin@mlhost:/media/shared/DeepDetect$
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:25
try killing dede and starting it with ./dede -host 0.0.0.0 and then try the curl command again
may want to stop the other docker container you have that's restarting itself (not sure if it's conflicting in some way)
rperdon
@rperdon
Oct 26 2017 19:29
it worked!
THANK YOU!
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:29
:D :+1:
rperdon
@rperdon
Oct 26 2017 19:31
So after all that effort, I get... a different result!
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:32
so the purpose of this was to compare the results from dd with this patch to digits i think, right?
rperdon
@rperdon
Oct 26 2017 19:35
So from what I can tell, the whole design was to put in BGR2RGB code for CV
the results came out different, but not correct
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:36
different compared to without the BGR2RGB code?
rperdon
@rperdon
Oct 26 2017 19:37
the results are different
cchadowitz-pf
@cchadowitz-pf
Oct 26 2017 19:43
ok, so the results from DD with the BGR2RGB code change are different from DIGITS?
rperdon
@rperdon
Oct 26 2017 19:45
yes
the bgr2rgb did not correct the divergence
So looking like back to square one for me for identifying the differences, in testing digits loading, I do notice a divergence in results, but not enough of a divergence to throw off the classification
Emmanuel Benazera
@beniz
Oct 26 2017 20:24
There's no mystery here, either the inputs are the same, either they are not. If you want us to help you debug more, you need to give us more exact information and allow us a way to reproduce. Typically, as previously suggested, this should start with a digit model you can share and a description of the exact output you are getting with digit and with DD for BGR and RGB.
rperdon
@rperdon
Oct 26 2017 21:59
I will get one prepared for tomorrow