Memory leak on Linux

[deadman2000]

Session close calling not released resources and tf threads are not stopped.

Example project demonstrated threads count increase: https://github.com/deadman2000/TFNetMemoryLeak

It's not TF bug, i tested on similar project on C: https://github.com/deadman2000/TFCThreadTest

It's Linux only problem. On Windows resources released success

[deadman2000]

I suggest root of problem in using ThreadLocal. Graph never disposed

[Oceania2018]

@deadman2000 You have to dispose Graph explictly.

[deadman2000]

Manually disposing Graph not helps

            session.graph.Dispose();
            session.close();

Test log:

2019-10-10 12:42:43.235019: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: model/
2019-10-10 12:42:43.238466: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-10-10 12:42:43.249303: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2019-10-10 12:42:43.276624: I tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: model/
2019-10-10 12:42:43.284454: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 49441 microseconds.
Dispose tf.Tensor '<unnamed Operation>' shape=(1,2) dtype=TF_FLOAT
Dispose Tensor disposing:True _disposed:False
  TF_DeleteTensor
Dispose grap-key-45/, (39260016)
Dispose Graph disposing:True _disposed:False
  TF_DeleteGraph
Dispose Tensorflow.Session
Dispose Session disposing:True _disposed:False
DisposeUnmanagedResources
  TF_DeleteSession
Dispose Tensorflow.Status
Dispose Status disposing:True _disposed:False
  TF_DeleteStatus
End
Threads: 71
Press Q to break or any another to repeat
2019-10-10 12:42:43.928763: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: model/
2019-10-10 12:42:43.932001: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2019-10-10 12:42:43.941843: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2019-10-10 12:42:43.969191: I tensorflow/cc/saved_model/loader.cc:151] Running initialization op on SavedModel bundle at path: model/
2019-10-10 12:42:43.973776: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 45020 microseconds.
Dispose tf.Tensor '<unnamed Operation>' shape=(1,2) dtype=TF_FLOAT
Dispose Tensor disposing:True _disposed:False
  TF_DeleteTensor
Dispose grap-key-49/, (38424432)
Dispose Graph disposing:True _disposed:False
  TF_DeleteGraph
Dispose Tensorflow.Session
Dispose Session disposing:True _disposed:False
DisposeUnmanagedResources
  TF_DeleteSession
Dispose Tensorflow.Status
Dispose Status disposing:True _disposed:False
  TF_DeleteStatus
End
Threads: 74
Press Q to break or any another to repeat

[Nucs]

This might also leak in Linux for an entirely different reason but for me on Windows;
Every time you run program (press any key in the console program in the repository) there is memory build up.

Only the following line causes leaking (with or without .as_default().

var session = Session.LoadFromSavedModel(modelLocation);`

Therefore the leak is within loading a saved model and disposing it later on.
@Oceania2018 To your attention.

[gosha20777]

I have the same problem with Linux

[ADH-LukeBollam]

I'm having a similar problem, I've got about 20 unit tests and memory isn't being completely returned after each one. I'm calling:
_session.graph.Dispose();
_session.Dispose();

I'm seeing slight growth after each one that loads a saved model

I am also using
Session.LoadFromSavedModel(modelLocation);

[Oceania2018]

@gosha20777 @LukeBolly Could you PR a minimul runnable code into https://github.com/SciSharp/TensorFlow.NET/tree/master/src/TensorFlowNet.Benchmarks/Leak?

[ADH-LukeBollam]

@Oceania2018 I haven't had time to create a repro for you, but I've been debugging another issue and have come across some stuff leftover from TensorFlow.NET in a memory dump after my TF process has finished and calling dispose on the graph and session

Confirming the objects are still in memory,
NumSharp

TensorFlow.NET

Could this be the UnmanagedMemoryBlocks not being released?

[Oceania2018]

@LukeBolly Could you run it in latest release? We've removed NumSharp dependency.

[ADH-LukeBollam]

Hi @Oceania2018, I've updated to the latest version. After running all of my unit tests and checking the managed memory at the end, there are still a large number of Tensorflow objects left in memory.

I've gone through my code and added Dispose() to all graphs, sessions, NDArrays, but I still end up with this locked up:

[ADH-LukeBollam]

While I can't share the model I'm using, loading and disposing it in a loop confirms there is an issue cleaning up resources when working with a SavedModel.

public class TestModel
{
        public TestModel(string classifierModelPath )
        {
	        for (var i = 0; i < 1000; i++)
	        {
		        var _classifierSession = Session.LoadFromSavedModel(classifierModelPath );
        
		        _classifierSession.graph.Exit();
		        _classifierSession.graph.Dispose();
		        _classifierSession.Dispose();
	        }
        }
}

[Oceania2018]

@LukeBolly Does it apply to any model? whether the other models cause the same issue?
Another way might be able to release the resource is tf.Context.reset_context().