smistad on v4.0
V4 refactor of PatchStitcher/Ge… (compare)
smistad on master
Update CI-windows.yml (compare)
smistad on master
Create CI-windows.yml (compare)
smistad on v4.0
Changed size_t to Py_ssize_t (compare)
smistad on master
Create CI-ubuntu.yml (compare)
smistad on v4.0
V4 refactor of all renderers Removed unused CMake code causi… V4 refactoring for NN. Fixed is… and 3 more (compare)
Hi guys, I've been looking into it for a while today, no luck so far.
I realized that the output of my segmentation model is zero everywhere. I tried changing it to a NeuralNetwork but, when I call .setInputConnection(), I get:
ERROR  Terminated with unhandled exception: NeuralNetwork has no input port with ID 0
I made no other changes to the code.
Anyway, I suspect that there's a problem with my model, so I'll try getting it to work with openVINO but outside of FAST first.
@gdims When running this example with pyFAST 3.2.0 on your machine, it rendered the US with predicted segmentation, right(?):
If yes, then there is likely something wrong with model and/or you are doing something different in preprocessing now than you did before/during training. What did you do in preprocessing when training the model? Perhaps you used standardization (Z-score normalization) instead of regular normalization 0-1?
I have done some progress. I successfully loaded an .onnx model after using opset <11.
To make the .setInputConnection() not throw the aforementioned error, I realized that for NeuralNetwork, you have to call .load() before .setInputConnection().
I then confirmed that the correct scaleFactor for my model is 1, since thresholding the output tensor at 0 gave a perfectly valid segmentation mask.
So, I now figured that I should pass this tensor output to a TensorToSegmentation() object, with a threshold of 0, and I'll be good to go.
Sadly, the output image of the TensorToSegmentation() object is still all zeros.
Is there something more I am missing here about TensorToSegmentation()? Does it expect the input to be in some range? My models raw output is somewhere in [-50, 50], very roughly.
[-50,50]? Did you forget to add an activation function at the end of the network? Seems like you are using a linear activation at the end, which is not optimal for performing semantic segmentation. If you are using a linear activation, you could try adding a softmax activation layer and then saving/exporting the frozen graph again.
That would also explain why FAST struggles to understand your predictions.
How did you train your model? Did you use PyTorch? It is quite common to produce the logits within the loss there. However, for deployment, I would recommend doing that within the graph
Hello again guys, have a nice week!
Exactly, I'm using PyTorch and my loss function contains the sigmoid. I might now change this according to what you are saying.
My thoughts on this are the following: since the activation function is not trainable, it should make no difference for the model's performance to include it in the graph.
I mean, the model learns the weights so that the sigmoided output maximizes DICE to ground truth. Assuming a 0.5 threshold for the sigmoid, we may just as well remove it and have a threshold of 0 for the raw logits (right before the sigmoid), and it should make no difference. Am I misunderstanding something fundamental?
That's right too, I only have one output channel as I'm doing binary segmentation and I thought it would be okay.
So I'll move my output to a second channel now and see what happens :)
Since you used a sigmoid in the loss function, appending a sigmoid to the graph, will not affect model performance. The model is trained using logits, so that shouldn't be an issue. Thresholding at 0 for the linear activation should also work fine, but I agree with @smistad that this might result in uncontrolled and odd behaviour during inference. Using sigmoid/softmax also makes the output more easily interpretable, especially for you case.
Nice! Glad you finally got it working :) Let us know if you have any future issues. Happy to help!
Alright, now this is a different issue.
I'm trying to run inference on the Intel Neural Compute Stick 2, which I have installed and successfully ran the included intel demos on.
However, in FAST I get the following:
INFO  OpenVINO: Node setup complete.
E: [ncAPI] [ 204881] [python] getFirmwarePath:637 Firmware not found in: /home/longjon/anaconda3/envs/echo/lib/python3.8/site-packages/fast/bin/../lib/../lib/../lib/usb-ma248x.mvcmd
E: [ncAPI] [ 204881] [python] ncDeviceOpen:915 Can't get firmware, error: NC_ERROR
INFO  Failed to get GPU/VPU plugin for OpenVINO inference engine: Can not init Myriad device: NC_MVCMD_NOT_FOUND
INFO  Trying CPU plugin instead..
INFO  OpenVINO: Inference plugin setup for device type CPU
INFO  OpenVINO: Network loaded.
So it doesn't have the firmware.
usb-ma248x.mvcmd does not exist in that directory.
Has anyone tried something like this?
Hey guys, I hope you're doing well!
I'd like to report one thing and ask another:
1) For ONNX models, exporting them with opset v11 gives issues (they can't be loaded in FAST). I tested this for deeplabv3 and fcn_resnet50 architectures. opsets v9 and v10 work ok.
2) I'd like to do some real-time processing on my segmentation output. I suppose Erik has faced this in the past. I have a model that is able to segment the LV endocardium (ENDO) and epicardium (EPI), and output the mask in different channels. If I want to obtain a segmentation of the myocardium only, I need to subtract ENDO from EPI. I tried including this procedure in the forward function of my model but such a model could only be exported with opset v11 in ONNX (the error is not clear). So, I guess I could do it post-inference.
So, given a model that outputs 2 segmentation masks, how would you go about obtaining a third mask, derived from the already available ones, and display it realtime?
Also, does FAST assume that multi-channel segmentation is two-channel only? My model currently outputs a Bx3xWxH tensor where B is the batch size, W and H are the image Width and Height.
Of the 3 channels, 1 is background, 2 is LV endocardium, 3 is LV epicardium.
However, for some reason, when I do:
segmentationRenderer.setColor(1, fast.Color.Red()) # ENDO
segmentationRenderer.setColor(2, fast.Color.Blue()) # EPI
, the two masks come out almost identical, both segmenting the epicardium. They are only different in some frames, at some pixel locations.
It's as if FAST cannot discriminate between the two outputs.
I have tested the model outside of fast and the outputs look correct.
Thanks for the quick replies Erik!
Oh, I understand. But shouldn't I be able to only render the endocardium mask by selecting it via setColor?
Anyway, then my model has to output 2 non-overlapping masks. I'm thinking, and maybe you can answer, it's better to train for epicardium and then subtract endocardium rather than train directly for myocardium, right? This way I can ensure that, where endocardium ends, the myocardium begins.
I'll try to see why my implementation of the above fails to export in ONNX format.
Another option is to split the tensor into two segmentation images and render them both
Regarding this, is there a way to do it (realtime) in Python? I've been doing some digging in the code and as far as I can understand, one has to go through C++ to get this level of control. Please correct me if I'm wrong. Good evening!