Input node?
tf.cond (i.e. if statements).
cos and sin of the timestamp. Is there a point to passing both to the model as input? Or do they only use one of them?
math.nan in place of sensor data but this broke training (the loss function returned nan). What should I do in this case? Set the values to zero? Set them to random values? Ideally I want the model to skip them and not use them for training.
Hello again,
We have gone from TF Java 0.2.0 to 0.3.1 on our linux hosts.
And basically just changed so that it compiles. We load and unload a bunch of different savedmodels.
We are now experiencing OutOfMemory on bytedeco as if the memory is not reclaimed. We try to calculate the size of the incoming savedmodel and only keep models to roughly half of the available memory of the host (we have a guava cache that get approximate weight of the model and only hold models up to half of the available memory).
We are closing all tensors, savedmodel-bundles etc (not really changing from 0.2.0).
-Dorg.bytedeco.javacpp.maxbytes=25G
-Dorg.bytedeco.javacpp.maxphysicalbytes=25GWe are about to try 0.4.0-SNAPSHOT and also do some kind of more specific test case.
23 replies
We have continued to try to nail down our problem with memory. We believe it is something strange with Linux version of 0.3.1 (and 0.4.0-SNAPSHOT). Some memory is not deallocated and it is quite visible when you run large models.
I’ve tried to build a “test case” for this. Maven project with a Dockerfile. The Dockerfile will download a somewhat large model from tfhub (efficientnet_b7_classification_1).
The test is basically:
for (int q = 0; q < 10; q++) {
//Download https://storage.googleapis.com/tfhub-modules/tensorflow/efficientnet/b7/classification/1.tar.gz
//and unpack inside src/main/resources
Path savedModelSourcePath = Path.of(Thread.currentThread().getContextClassLoader().getResource("efficientnet_b7_classification_1").getFile());
try (SavedModelBundle savedModelBundle = SavedModelBundle.load(savedModelSourcePath.toString(), "serve")) {
System.out.printf("After loading model, physicalBytes: %d, totalBytes: %d\n", Pointer.physicalBytes(), Pointer.totalBytes());
}
Pointer.deallocateReferences(); //trying to force
System.gc(); //trying to force
}When running on Linux, memory will increase and usually crash after 3-4 iterations (depending on bytedeco mem flags or host memory) but running same test on OSX test will usually pass and memory decreases from time to time.
Do you think this is related to tensorflow/java#304 or something else?
We are trying to do some more with Tensorflow debug-logging with allocation and deallocation to try to find some pointers where this could be.
Hi, I'm also investigating this memory leak together with @jakeri.
I'm not sure i completely grok all the interactions between tensorflow, javacpp and the java wrappers but it looks like the TF_Session handle isn't destroyed when closing the Session. This can be confirmed by running with tensorflow VLOG set to 2 which will log allocations and deallocations. Some things are never deallocated (memory allocated restore ops for example).
If I understand things correctly Session::delete will be invoked when closing the session, which should release the reference and ultimately the Pointer should be deallocated, which should trigger the the DeleteDeallocator in AbstractTF_Session. This works when calling TF_Session::newSession since that will set the deallocator on the pointer before returning, but, when loading a saved model the TF_Session object is created internally in native code in TF_LoadSessionFromSavedModel and returned, which means the deallocator is never set on the Pointer, which means the DeleteDeallocator is never invoked.
When manually calling TF_CloseSession and TF_DeleteSession on the native handles it leaks much less memory (still a little it I think, but significantly less).
input_image=name: "serving_default_input_image:0"
dtype: DT_FLOAT
tensor_shape {
dim {
size: -1
}
dim {
size: -1
}
dim {
size: -1
}
dim {
size: 3
}
}
@JamesPiggott , I don’t know if you are subscribed to the TensorFlow discussion forum but it is the new TF platform for starting new discussion and we’ll slowly migrate the different topics started on this Gitter channel to it.
That being said, I’ve replied and posted an example of converting BufferedImage instances to a float tensor (TFloat32) in this post, if you can please take a look and maybe continue the discussion from there?
(BTW sorry but my original snippet was in Kotlin, let me know if you need help to convert it to Java)