Segmentation fault in multithread app (v0.11.2)

[deadman2000]

App is crashing during sess.run with message Segmentation fault (core dumped) on docker or Attempted to read or write protected memory. This is often an indication that other memory is corrupt. on Windows.

Stack trace on Windows

   at Tensorflow.c_api.TF_SessionRun(IntPtr session, TF_Buffer* run_options, TF_Output[] inputs, IntPtr[] input_values, Int32 ninputs, TF_Output[] outputs, IntPtr[] output_values, Int32 noutputs, IntPtr[] target_opers, Int32 ntargets, IntPtr run_metadata, IntPtr status)
   at Tensorflow.BaseSession._call_tf_sessionrun(KeyValuePair`2[] feed_dict, TF_Output[] fetch_list, List`1 target_list)
   at Tensorflow.BaseSession._do_run(List`1 target_list, List`1 fetch_list, Dictionary`2 feed_dict)
   at Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict)
   at Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict)

I created small example project for tests: https://github.com/deadman2000/TensorFlowNetMultithreading

[Nucs]

Task.Run does not promise a different thread will be used for every created Task, in-fact it is more likely that the tasks are queued behind 4 to 8 Threads (depending on your CPU).
The default mechanism in Tensorflow.NET uses ThreadID.
Replace Task.Run with new Thread(()=> ...).Start(); or create a long-running task using task factory.

Also LoadFromSavedModel() does not set the created Session as default (@Oceania2018 it should, fix it please)
Replace the LoadFromSavedModel to the following:
_session = Session.LoadFromSavedModel(modelLocation).as_default();

Let me know if that made any difference.

[Nucs]

I've tested it locally and it works fluently.
Program.cs: https://pastebin.com/upNWzmHk
Predictor.cs: https://pastebin.com/i96ZMyVV

I'll add information about it the wiki.

[Nucs]

If you'll need to do multi-threaded unit tests in the future, you are welcome to use MultiThreadedUnitTestExecuter I wrote for the library:
https://github.com/SciSharp/TensorFlow.NET/blob/master/test/TensorFlowNET.UnitTest/Utilities/MultiThreadedUnitTestExecuter.cs

Usage:

MultiThreadedUnitTestExecuter.Run(threadCount: 8, worload: tid => ...);

[deadman2000]

@Nucs as_default not helps. Still Segmentation fault and AccessViolationException with random errors:

2019-09-10 10:20:22.989570: E tensorflow/core/framework/types.cc:102] Unrecognized DataType enum value 105
Expects arg[1] to be float but unknown dtype enum (105)_ref is provided
Tensorflow.TensorflowException: Expects arg[1] to be float but unknown dtype enum (105)_ref is provided
   в Tensorflow.Status.Check(Boolean throwException)
   в Tensorflow.BaseSession._call_tf_sessionrun(KeyValuePair`2[] feed_dict, TF_Output[] fetch_list, List`1 target_list)
   в Tensorflow.BaseSession._do_run(List`1 target_list, List`1 fetch_list, Dictionary`2 feed_dict)
   в Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict)
   в Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict)
   в CalcEventsTFS.Predictor.Predict(Single[][] inputs)
   в CalcEventsTFS.Program.<>c.<Main>b__2_0()
Expects arg[1] to be float but bfloat16 is provided
Tensorflow.TensorflowException: Expects arg[1] to be float but bfloat16 is provided
   в Tensorflow.Status.Check(Boolean throwException)
   в Tensorflow.BaseSession._call_tf_sessionrun(KeyValuePair`2[] feed_dict, TF_Output[] fetch_list, List`1 target_list)
   в Tensorflow.BaseSession._do_run(List`1 target_list, List`1 fetch_list, Dictionary`2 feed_dict)
   в Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict)
   в Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict)
   в CalcEventsTFS.Predictor.Predict(Single[][] inputs)
   в CalcEventsTFS.Program.<>c.<Main>b__2_0()
Expects arg[1] to be float but resource_ref is provided
Tensorflow.TensorflowException: Expects arg[1] to be float but resource_ref is provided
   в Tensorflow.Status.Check(Boolean throwException)
   в Tensorflow.BaseSession._call_tf_sessionrun(KeyValuePair`2[] feed_dict, TF_Output[] fetch_list, List`1 target_list)
   в Tensorflow.BaseSession._do_run(List`1 target_list, List`1 fetch_list, Dictionary`2 feed_dict)
   в Tensorflow.BaseSession._run(Object fetches, FeedItem[] feed_dict)
   в Tensorflow.BaseSession.run(Tensor fetche, FeedItem[] feed_dict)
   в CalcEventsTFS.Predictor.Predict(Single[][] inputs)
   в CalcEventsTFS.Program.<>c.<Main>b__2_0()

[Nucs]

I've fixed it in the commit above, untill it is available via nuget,
at the Predictor constructor, add a lock in the following manner:

lock (Locks.ProcessWide)
    _session = Session.LoadFromSavedModel(modelLocation).as_default();

As mentioned in the wiki, due to lack of documentation from TF's side - we don't know what APIs are not threadsafe and it appears that c_api.TF_LoadSessionFromSavedModel() is not.

[deadman2000]

With lock on .Net Framework 4.6.1 random errors.
On .Net Core 2.2 Segmentation fault and AccessViolationException

I found out that the problem is related to garbage collection.

Try this example:

for (int t = 0; t < THREADS_COUNT; t++)
{
    new Thread(() =>
    {
        Session sess;

        lock (Locks.ProcessWide)
            sess = Session.LoadFromSavedModel(modelLocation).as_default();

        {
            var inputs = new[] { "sp", "fuel" };

            var inp = inputs.Select(name => sess.graph.OperationByName(name).output).ToArray();
            var outp = sess.graph.OperationByName("softmax_tensor").output;

            for (var i = 0; i < 1000; i++)
            {
                {
                    var data = new float[96];
                    FeedItem[] feeds = new FeedItem[2];

                    for (int f = 0; f < 2; f++)
                        feeds[f] = new FeedItem(inp[f], new NDArray(data));

                    try
                    {
                        sess.run(outp, feeds);
                    }
                    catch (Exception ex)
                    {
                        Console.WriteLine(ex);
                    }
                }
                GC.Collect();
            }
        }
    }).Start();
}

or test project https://github.com/deadman2000/TensorFlowNetMultithreading

[Nucs]

Please try the following code:
https://pastebin.com/EL9Cv6FQ
It has worked for me for over 10 runs both in Debug and Release.

[deadman2000]

Yes, this sample works fine. But after i replaced cycle to infinity loop, it has crashed after 1 minute in Debug and after 30 seconds in Release

[Nucs]

I ran it for over 35 minutes (and still running) without a crash, the while(true) was set on the sess.run scope and not the entire script.
Second run worked for 20 minutes. this time I included the OperationByName calls in the loop and didn't use the VS debugger.

This has lead me to believe it is a game of chance and indeed the program crashed eventually after 10-60 seconds with the following message in Output Window

The program '[22268] dotnet.exe' has exited with code -1073740791 (0xc0000409).

When an error is a simple exit code it indicates that C++ (Tensorflow) has caused the crash.

Code -1073740791 (0xc0000409).
Stack buffer overflow / overrun. Error can indicate a bug in the executed software that causes stack overflow, leading to abnormal termination of the software.

Even though we do not call the same Session from different threads.
So it must be something complicated that is worth Tensorflow's team attention.

I'll try to create a memory dump and see if we can indicate more accurately where the problem lies.

[Oceania2018]

@Nucs Should we raise this issue to tensorflow team?

[Nucs]

After I'll get my hands on a dump and research it. I'll let you know.