ObjectDisposedException on shutdown

[Tratcher]

An app running in IIS with OWIN integration gets ObjectDisposedExceptions on shutdown rather than gracefully draining requests.

Exception Details: System.ObjectDisposedException, Cannot access a disposed object.
Object name: '[Redacted].AuthenticationHandler'.,
    at System.Net.Http.DelegatingHandler.CheckDisposed()
   at System.Net.Http.DelegatingHandler.SetOperationStarted()
   at System.Net.Http.DelegatingHandler.SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
   at [Redacted].AuthenticationHandler.<SendAsync>d__12.MoveNext()

Here's where dispose is called:

   at [Redacted].AuthenticationHandler.Dispose(Boolean disposing)
   at System.Net.Http.DelegatingHandler.Dispose(Boolean disposing)
   at System.Web.Http.HttpServer.Dispose(Boolean disposing)
   at System.Net.Http.HttpMessageInvoker.Dispose(Boolean disposing)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.OnAppDisposing()
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.CancellationCallbackInfo.ExecuteCallback()
   at System.Threading.CancellationTokenSource.ExecuteCallbackHandlers(Boolean throwOnFirstException)
   at System.Threading.CancellationTokenSource.NotifyCancellation(Boolean throwOnFirstException)
   at Microsoft.Owin.Host.SystemWeb.ShutdownDetector.Cancel()
   at Microsoft.Owin.Host.SystemWeb.ShutdownDetector.Stop(Boolean immediate)
   at System.Web.Hosting.HostingEnvironment.StopRegisteredObjects(Boolean immediate)
   at System.Web.Hosting.HostingEnvironment.InitiateShutdownWorkItemCallback(Object state)
   at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
   at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
   at System.Threading.ThreadPoolWorkQueue.Dispatch()'

The root event here is HostingEnvironment.StopListening. "Occurs when the worker process or application pool associated with this host has stopped listening for new requests and will eventually shut down." The app is shutting down, but there may still be requests in flight that should be completed if possible.

Microsoft.Owin wired into the StopListening event so that SignalR could gracefully shut down long polling and WebSocket requests. WebAPI however wired up it's pipeline disposal to this event. This causes the pipeline to be disposed while request are in flight.

AspNetWebStack/src/System.Web.Http.Owin/WebApiAppBuilderExtensions.cs

Line 111 in ba26cfb

AppDisposing = builder.GetOnAppDisposingProperty()

AspNetWebStack/src/System.Web.Http.Owin/WebApiAppBuilderExtensions.cs

Lines 119 to 128 in ba26cfb

	IDictionary<string, object> properties = builder.Properties;
	if (properties == null)
	{
	return CancellationToken.None;
	}
	object value;
	if (!properties.TryGetValue("host.OnAppDisposing", out value))

https://github.com/aspnet/AspNetWebStack/blame/ba26cfbfbf958d548e4c0a96e853250f13450dc6/src/System.Web.Http.Owin/HttpMessageHandlerAdapter.cs#L87

There is no alternate event in IIS that fires when requests have drained.

Workaround: Remove the CancellationToken from the IAppBuilder so WebAPI cannot register with it.

appBuilder.Properties.Remove("host.OnAppDisposing");

Proposed fix: WebApi should not register for disposal with this event.
https://github.com/aspnet/AspNetWebStack/blame/ba26cfbfbf958d548e4c0a96e853250f13450dc6/src/System.Web.Http.Owin/HttpMessageHandlerAdapter.cs#L87
(When should it be disposed then?)

[mkArtakMSFT]

@Tratcher wouldn't the propose fix be a breaking change? What will be the downsides of it?

[Tratcher]

The handlers are disposed too soon, causing unnecessary failures. We'd have to figure out a better signal to dispose them before answering questions about breaking changes.

[Bykiev]

Is any chance it will be ever fixed?

[k290]

Hit this bug in our app in prod too during a deployment.