Using System.Transaction with IStatelessSession doesn't always flush batches to database

[jklawrence]

I'm having two issues with using System.Transaction and IStatelessSession together.

First, it doesn't work at all with the default UseConnectionOnSystemTransactionPrepare value of true. Committing a transaction like:

using (var transaction = new TransactionScope())
{
	using (var session = sessionFactory.OpenStatelessSession())
	{
		// save a change here
	}
	transaction.Complete();
}

always results in this exception:

System.Transactions.TransactionAbortedException: The transaction has aborted. ---> System.InvalidOperationException: Batcher still has opened ressources at time of processing from system transaction.
   at NHibernate.AdoNet.ConnectionManager.BeginProcessingFromSystemTransaction(Boolean allowConnectionUsage)
   at NHibernate.Transaction.AdoNetWithSystemTransactionFactory.SystemTransactionContext.Prepare(PreparingEnlistment preparingEnlistment)
   --- End of inner exception stack trace ---
   at System.Transactions.TransactionStateAborted.EndCommit(InternalTransaction tx)
   at System.Transactions.CommittableTransaction.Commit()
   at System.Transactions.TransactionScope.InternalDispose()
   at System.Transactions.TransactionScope.Dispose()
   at StatelessBatchFlush.Program.ValidateBatching(ISessionFactory sessionFactory) in C:\Projects\NHibernate-Stateless\Program.cs:line 68

Setting UseConnectionOnSystemTransactionPrepare=false does cause this exception to go away, but with that set the last batch of items does not get flushed to the database. IStatelessSession does not expose a Flush() method, but we've found that calling session.GetSessionImplementation().Flush() before completing the transaction is a workaround for this.

I've created a small console app to demo the problems I'm having here: https://github.com/jklawrence/NHibernate-Stateless

These same scenarios work fine when using an NHibernate transaction instead of System.Transaction.

NHibernate version: 5.3.8

[fredericDelaporte]

When using a NHibernate transaction, is it truly the same scenario? A Nhibernate transaction cannot be opened before opening the session, and must be committed (or rollbacked) before closing the session.

So when using a NHibernate transaction, the nesting of operations is inverted: open session > open transaction > close transaction > close session.
While with the transaction scope, your code is currently doing this: open transaction > open session > close session > close transaction.

Committing the transaction, with default settings, triggers a flush. That is why there is no issues with a NHibernate transaction.

I think the trouble here is not about UseConnectionOnSystemTransactionPrepare setting. The trouble is the stateless session somehow "forget" to flush the last changes when getting closed. A stateless session, by essence, is not supposed to keep changes in memory for flushing them later. But with batching enabled, it can still use a batcher which will do that. Performance wise, it is a good thing for some scenario, but it seems it is not handled right currently.

There are other troubles caused by that, as shown by this Stack Overflow question: it may cause a query following some changes to not yield back the changed data, due to it not being already flushed to database.

I think we need to add some checks in the stateless session to ensure it flushes its batcher when doing some other queries or when getting closed. An easier fix could be to not let the stateless session use a batcher, but that would cause performance regressions for some usages.

As a workaround for your case, either you can nest your transaction scope operations inside your session lifetime, or you should explicitly flush the stateless session batcher by using session.GetSessionImplementation().Flush().

(I have not checked if the stateless session will correctly enlist its connection into a transaction scope opened after the session. Regular sessions with the default transaction factory will do that, but I do not remember about stateless ones.)

[hazzik]

From what I see in the code the default and only supported flush mode of the stateless session is Commit. This flush mode stipulates explicit commit of a transaction by the user. But there is no transaction because you are not participating in any (UseConnectionOnSystemTransactionPrepare=false). So it is sort-of "by design".

So, I would disagree with @fredericDelaporte on this:

I think the trouble here is not about UseConnectionOnSystemTransactionPrepare setting.

[hazzik]

@fredericDelaporte when I comment these lines of code everything starts working:

nhibernate-core/src/NHibernate/AdoNet/ConnectionManager.cs

Lines 552 to 553 in 07ecb9f

	if (Batcher.HasOpenResources)
	throw new InvalidOperationException("Batcher still has opened ressources at time of processing from system transaction.");

I'm wondering is this check even needed?

[hazzik]

Sessions put all pending operations into so called ActionQueue, while StatelessSession just puts them into a batcher. So the check above almost never makes sense for Session because at the time of the queue the batcher is empty and all pending operations are in the action queue. On flush the pending operations will be sorted and then executed by the batcher. On the other hand this check will always fail for the stateless session if it is not flushed.

[fredericDelaporte]

The batcher being not empty there means it has commands initialized with the current connection, and so the connection swap does not affect it and we cannot guarantee flushing it will not cause a concurrent usage of its connection with the connection own 2PC, in distributed cases.
That is the reason for this check, and it is there to detect we have a trouble. From my viewpoint, it is doing its job.
We need to flush the batcher before arriving in the 2PC, or we are in a bad situation: possible multi-threaded usage of the same connection.

So the check above almost never makes sense for Session

This check is a safeguard against a situation which should not occur, otherwise it is a potential threading bug. So if the session is not bugged, this check should never trigger. It means the stateless session is bugged : it leaves a batcher with pending operations on the connection opened after being closed and upon entering its 2PC.

From what I see in the code the default and only supported flush mode of the stateless session is Commit.
This flush mode stipulates explicit commit of a transaction by the user. But there is no transaction because you are not participating in any (UseConnectionOnSystemTransactionPrepare=false). So it is sort-of "by design".

That is not the meaning of UseConnectionOnSystemTransactionPrepare. This setting does not prevent participating in a transaction scope. It prevents using the "enlist additional resources during first phase of 2PC" feature, which is required for "flush on commit" mode to work with transaction scopes. So essentially, this setting disable FlushMode.Commit with transaction scope, as documented here.

As of NHibernate v5.0, FlushMode.Commit requires the configuration setting transaction.use_connection_on_system_prepare to be true for flushing from transaction scope commit. Otherwise, it will be your responsibility to flush the session before completing the scope.

Following paragraph then explain why we should avoid using it.

So we have a trouble: due to risks of (nasty threading) bugs when trying to use FlushMode.Commit with transaction scopes, we recommend not using it and flushing the session explicitly instead before completing the scope. But the stateless session does not exposes a Flush.

First I think that leaving a batcher non empty after closing a stateless session is a bug: it should clean-up, which for its case means flushing the batcher.
The stateless session is not in FlushMode.Commit, it is stateless: it normally flushes immediately. But when batching is enabled, it is no more fully stateless, it has the batcher with some state, and for avoiding this batching feature of stateless sessions to turn into just a big bug, it is flushed on Commit. But that is not enough indeed. It should also be flushed on closing and on any other querying operations.

That would by the way solves this issue reported here, where the scope surround the session life-cycle.

But if the session life-cycle surround the scope, that will very likely still crash or lose some changes or do them out of expected transaction. For fixing this, there is no other solution than an explicit flush, so we will need to exposes something like a FlushBatcher method, that users will have to call if they use batching with stateless session and transaction scopes completed during the lifetime of the session.

[gokhanabatay]

Is it related ? #2309

[fredericDelaporte]

Not really. Well is is somehow related because the trouble here is also due to the batcher usage in the stateless session being a bit messy.
But it is not related enough for fixing #2309 while fixing #2750.