Stack overflow in promise chain when performing web scraping.

[timprepscius]

I've written a little scraper which can use both the foundation URL class and the HTTPClient classes. Requests go through many different proxies pulled from proxy lists on the web.

The HTTPClient classes are failing.
Do you want bugs yet, or should I check back in a couple months?

[ianpartridge]

Bugs please!

[timprepscius]

Ok, usually when I put bug information, I can provide specifics, but the world of proxy-ing is pretty much a black box to me, and I'm not actually running any proxies myself.

That being said, I took my code and distilled it, and made this very basic test.
https://github.com/timprepscius/SimpleProxy

It will get 300 or so proxies from a few different lists.
Then it will download "https://code.jquery.com/jquery-3.4.1.slim.js" using NIO and Foundation for each proxy.

The foundation code is going to pop up a dialog which you have to press ok on, many times. (depends on the proxies)- which is the whole reason I want to incorporate the nio code, to get rid of this dialog.

Anyhow, you will see that some of the results are not the same.
NIO fails sometimes which Foundation doesn't. And in my real application, NIO seems to get stuck every once in a while (haven't seen that in this tester so far).

Don't know if this is helpful or not.

[timprepscius]

An additional test path, for the stuck-ness, might be to use NIO against a proxy against a cloudflare rate limited website. (and trigger the rate limiting)

Foundation correctly bails - I think NIO is getting honeypotted.

[Lukasa]

When you say "some of the results are not the same", can you elaborate on the differences? Are they consistent for the same proxies, or do different proxies trigger this each time?

[timprepscius]

I thought I was seeing that Foundation would sometimes succeed in using a proxy, where nio would fail. But I'm not so sure now. It could have been the proxy being unstable. Will need to run tester more to find out.

In the mean time, I let the scraper run last night with NIO and woke up to this stack trace.
Will try to attach Ahh don't know how to attach.

It looks like this

thread #74, name = 'NIO-ELT-#0', stop reason = EXC_BAD_ACCESS (code=2, address=0x70000ea39ff8)
    frame #0: 0x00007fff76335b4c libswiftCore.dylib`swift_getGenericMetadata + 12
    frame #1: 0x00007fff76306870 libswiftCore.dylib`merged type metadata accessor for Swift.LazyMapSequence + 32
    frame #2: 0x00007fff762da1e0 libswiftCore.dylib`type metadata accessor for Swift.Result + 16
    frame #3: 0x000000010099d5ef NIO`EventLoopFuture._setValue(value=failure, self=<unavailable>) at EventLoopFuture.swift:0:17 [opt]
  * frame #4: 0x000000010099ca0b NIO`EventLoopPromise._resolve(value:) [inlined] NIO.EventLoopPromise._setValue(value=Swift.Result<τ_0_0, Error> @ 0x000070000ea3a160, self=NIO.EventLoopPromise<τ_0_0> @ scalar) -> NIO.CallbackList at EventLoopFuture.swift:226:34 [opt]
    frame #5: 0x000000010099c9f7 NIO`EventLoopPromise._resolve(value=failure, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:211 [opt]
    frame #6: 0x0000000100999fca NIO`EventLoopPromise.fail(error=NIO.IOError @ 0x0000000107299f08, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:184:14 [opt]
    frame #7: 0x00000001009a268d NIO`closure #1 in EventLoopFuture.cascade(result=failure, promise=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ 0x000070000ea3a2a0) at EventLoopFuture.swift:817:47 [opt]
    frame #8: 0x00000001009a88d1 NIO`partial apply for closure #1 in EventLoopFuture.cascade(to:) at <compiler-generated>:0 [opt]
    frame #9: 0x00000001009a1942 NIO`closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () at EventLoopFuture.swift:709:13
    frame #10: 0x00000001009b54b5 NIO`merged partial apply forwarder for closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () + 21
    frame #11: 0x00000001009a88b0 NIO`partial apply forwarder for closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () + 16
    frame #12: 0x000000010098f54f NIO`thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) at <compiler-generated>:0 [opt]
    frame #13: 0x00000001009b1971 NIO`partial apply for thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) at <compiler-generated>:0 [opt]
    frame #14: 0x00000001009b6439 NIO`reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) to @escaping @callee_guaranteed () -> (@out NIO.CallbackList)partial apply forwarder with unmangled suffix ".697" + 9
    frame #15: 0x00000001009a6b85 NIO`specialized CallbackList._run() [inlined] reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out NIO.CallbackList) to @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) at <compiler-generated>:0 [opt]
    frame #16: 0x00000001009a6b7b NIO`specialized CallbackList._run() at EventLoopFuture.swift:109 [opt]
    frame #17: 0x000000010099ca2a NIO`EventLoopPromise._resolve(value:) [inlined] NIO.CallbackList._run() -> () at <compiler-generated>:0 [opt]
    frame #18: 0x000000010099ca1c NIO`EventLoopPromise._resolve(value=failure, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:211 [opt]
    frame #19: 0x0000000100999fca NIO`EventLoopPromise.fail(error=NIO.IOError @ 0x0000000107299f08, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:184:14 [opt]
    frame #20: 0x00000001009a268d NIO`closure #1 in EventLoopFuture.cascade(result=failure, promise=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ 0x000070000ea3a710) at EventLoopFuture.swift:817:47 [opt]
    frame #21: 0x00000001009a88d1 NIO`partial apply for closure #1 in EventLoopFuture.cascade(to:) at <compiler-generated>:0 [opt]
    frame #22: 0x00000001009a1942 NIO`closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () at EventLoopFuture.swift:709:13
    frame #23: 0x00000001009b54b5 NIO`merged partial apply forwarder for closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () + 21
    frame #24: 0x00000001009a88b0 NIO`partial apply forwarder for closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () + 16
    frame #25: 0x000000010098f54f NIO`thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) at <compiler-generated>:0 [opt]
    frame #26: 0x00000001009b1971 NIO`partial apply for thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) at <compiler-generated>:0 [opt]
    frame #27: 0x00000001009b6439 NIO`reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) to @escaping @callee_guaranteed () -> (@out NIO.CallbackList)partial apply forwarder with unmangled suffix ".697" + 9
    frame #28: 0x00000001009a6b85 NIO`specialized CallbackList._run() [inlined] reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out NIO.CallbackList) to @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) at <compiler-generated>:0 [opt]
    frame #29: 0x00000001009a6b7b NIO`specialized CallbackList._run() at EventLoopFuture.swift:109 [opt]
    frame #30: 0x000000010099ca2a NIO`EventLoopPromise._resolve(value:) [inlined] NIO.CallbackList._run() -> () at <compiler-generated>:0 [opt]
    frame #31: 0x000000010099ca1c NIO`EventLoopPromise._resolve(value=failure, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:211 [opt]
    frame #32: 0x0000000100999fca NIO`EventLoopPromise.fail(error=NIO.IOError @ 0x0000000107299f08, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:184:14 [opt]
    frame #33: 0x00000001009a268d NIO`closure #1 in EventLoopFuture.cascade(result=failure, promise=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ 0x000070000ea3ab80) at EventLoopFuture.swift:817:47 [opt]
    frame #34: 0x00000001009a88d1 NIO`partial apply for closure #1 in EventLoopFuture.cascade(to:) at <compiler-generated>:0 [opt]
    frame #35: 0x00000001009a1942 NIO`closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () at EventLoopFuture.swift:709:13
    frame #36: 0x00000001009b54b5 NIO`merged partial apply forwarder for closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () + 21
    frame #37: 0x00000001009a88b0 NIO`partial apply forwarder for closure #1 () -> NIO.CallbackList in NIO.EventLoopFuture.whenComplete((Swift.Result<A, Swift.Error>) -> ()) -> () + 16
    frame #38: 0x000000010098f54f NIO`thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) at <compiler-generated>:0 [opt]
    frame #39: 0x00000001009b1971 NIO`partial apply for thunk for @escaping @callee_guaranteed () -> (@owned CallbackList) at <compiler-generated>:0 [opt]
    frame #40: 0x00000001009b6439 NIO`reabstraction thunk helper from @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) to @escaping @callee_guaranteed () -> (@out NIO.CallbackList)partial apply forwarder with unmangled suffix ".697" + 9
    frame #41: 0x00000001009a6b85 NIO`specialized CallbackList._run() [inlined] reabstraction thunk helper from @escaping @callee_guaranteed () -> (@out NIO.CallbackList) to @escaping @callee_guaranteed () -> (@owned NIO.CallbackList) at <compiler-generated>:0 [opt]
    frame #42: 0x00000001009a6b7b NIO`specialized CallbackList._run() at EventLoopFuture.swift:109 [opt]
    frame #43: 0x000000010099ca2a NIO`EventLoopPromise._resolve(value:) [inlined] NIO.CallbackList._run() -> () at <compiler-generated>:0 [opt]
    frame #44: 0x000000010099ca1c NIO`EventLoopPromise._resolve(value=failure, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:211 [opt]

continuing, repeating in the same fashion til

    frame #5966: 0x00000001009a7764 NIO`specialized CallbackList._run() at EventLoopFuture.swift:83:27 [opt]
    frame #5967: 0x000000010099ca2a NIO`EventLoopPromise._resolve(value:) [inlined] NIO.CallbackList._run() -> () at <compiler-generated>:0 [opt]
    frame #5968: 0x000000010099ca1c NIO`EventLoopPromise._resolve(value=failure, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:211 [opt]
    frame #5969: 0x0000000100999fca NIO`EventLoopPromise.fail(error=NIO.IOError @ 0x0000000107299f08, self=NIO.EventLoopPromise<AsyncHTTPClient.HTTPClient.Response> @ scalar) at EventLoopFuture.swift:184:14 [opt]
    frame #5970: 0x00000001004a301a AsyncHTTPClient`TaskHandler.errorCaught(context:error:) [inlined] AsyncHTTPClient.HTTPClient.Task.fail(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) -> () at HTTPHandler.swift:405:26 [opt]
    frame #5971: 0x00000001004a300a AsyncHTTPClient`TaskHandler.errorCaught(error=NIO.IOError @ 0x0000000107299f08, self=0x00000001073a1c80) at HTTPHandler.swift:615 [opt]
    frame #5972: 0x00000001004a329e AsyncHTTPClient`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance TaskHandler<A> at <compiler-generated>:0 [opt]
    frame #5973: 0x000000010095e6ab NIO`ChannelHandlerContext.invokeErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) at ChannelPipeline.swift:1368:28 [opt]
    frame #5974: 0x000000010094bce9 NIO`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance ServerBootstrap.AcceptHandler [inlined] NIO.ChannelHandlerContext.fireErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) -> () at ChannelPipeline.swift:1168:20 [opt]
    frame #5975: 0x000000010094bcde NIO`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance ServerBootstrap.AcceptHandler [inlined] function signature specialization <Arg[2] = Dead> of (context=<unavailable>, error=NIO.IOError @ 0x0000000107299f08):NIO._ChannelInboundHandler.errorCaught(context: NIO.ChannelHandlerContext, error: Swift.Error) -> () at ChannelHandler.swift:308 [opt]
    frame #5976: 0x000000010094bcc4 NIO`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance ServerBootstrap.AcceptHandler [inlined] (extension in NIO):NIO._ChannelInboundHandler.errorCaught(context: NIO.ChannelHandlerContext, error: Swift.Error) -> () at <compiler-generated>:0 [opt]
    frame #5977: 0x000000010094bcc4 NIO`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance ServerBootstrap.AcceptHandler at <compiler-generated>:307 [opt]
    frame #5978: 0x000000010094e1d9 NIO`protocol witness for NIO._ChannelInboundHandler.errorCaught(context: NIO.ChannelHandlerContext, error: Swift.Error) -> () in conformance NIO.IdleStateHandler : NIO._ChannelInboundHandler in NIO + 9
    frame #5979: 0x000000010095e6ab NIO`ChannelHandlerContext.invokeErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) at ChannelPipeline.swift:1368:28 [opt]
    frame #5980: 0x000000010094bd39 NIO`_ChannelInboundHandler.errorCaught(context:error:) [inlined] NIO.ChannelHandlerContext.fireErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) -> () at ChannelPipeline.swift:1168:20 [opt]
    frame #5981: 0x000000010094bd2e NIO`_ChannelInboundHandler.errorCaught(context:error:) [inlined] function signature specialization <Arg[2] = Dead> of (context=<unavailable>, error=NIO.IOError @ 0x0000000107299f08):NIO._ChannelInboundHandler.errorCaught(context: NIO.ChannelHandlerContext, error: Swift.Error) -> () at ChannelHandler.swift:308 [opt]
    frame #5982: 0x000000010094bd14 NIO`_ChannelInboundHandler.errorCaught(context:error:) at <compiler-generated>:0 [opt]
    frame #5983: 0x0000000100495b39 AsyncHTTPClient`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance HTTPClientProxyHandler at <compiler-generated>:0 [opt]
    frame #5984: 0x000000010095e6ab NIO`ChannelHandlerContext.invokeErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) at ChannelPipeline.swift:1368:28 [opt]
    frame #5985: 0x000000010094bd39 NIO`_ChannelInboundHandler.errorCaught(context:error:) [inlined] NIO.ChannelHandlerContext.fireErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) -> () at ChannelPipeline.swift:1168:20 [opt]
    frame #5986: 0x000000010094bd2e NIO`_ChannelInboundHandler.errorCaught(context:error:) [inlined] function signature specialization <Arg[2] = Dead> of (context=<unavailable>, error=NIO.IOError @ 0x0000000107299f08):NIO._ChannelInboundHandler.errorCaught(context: NIO.ChannelHandlerContext, error: Swift.Error) -> () at ChannelHandler.swift:308 [opt]
    frame #5987: 0x000000010094bd14 NIO`_ChannelInboundHandler.errorCaught(context:error:) at <compiler-generated>:0 [opt]
    frame #5988: 0x000000010094c589 NIO`protocol witness for _ChannelInboundHandler.errorCaught(context:error:) in conformance ByteToMessageHandler<A> at <compiler-generated>:0 [opt]
    frame #5989: 0x000000010095e6ab NIO`ChannelHandlerContext.invokeErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) at ChannelPipeline.swift:1368:28 [opt]
    frame #5990: 0x000000010095e6e0 NIO`ChannelHandlerContext.invokeErrorCaught(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) at ChannelPipeline.swift:1370:24 [opt]
    frame #5991: 0x0000000100994e7d NIO`specialized SelectableEventLoop.handleEvent<A>(_:channel:) [inlined] NIO.ChannelPipeline.fireErrorCaught0(error=NIO.IOError @ 0x0000000107299f08, self=<unavailable>) -> () at ChannelPipeline.swift:859:29 [opt]
    frame #5992: 0x0000000100994e71 NIO`specialized SelectableEventLoop.handleEvent<A>(_:channel:) [inlined] generic specialization <NIO.Socket> of NIO.BaseSocketChannel.(self=<unavailable>)() -> NIO.BaseSocketChannel<A>.(ReadStreamState in _9C3D3FC8F9F3D83A05D3F6063F6AE958) at BaseSocketChannel.swift:986 [opt]
    frame #5993: 0x0000000100994e71 NIO`specialized SelectableEventLoop.handleEvent<A>(_:channel:) [inlined] generic specialization <NIO.Socket> of NIO.BaseSocketChannel.readEOF0(self=<unavailable>) -> () at BaseSocketChannel.swift:889 [opt]
    frame #5994: 0x0000000100994e71 NIO`specialized SelectableEventLoop.handleEvent<A>(_:channel:) [inlined] generic specialization <NIO.Socket> of NIO.BaseSocketChannel.reset(self=<unavailable>) -> () at BaseSocketChannel.swift:913 [opt]
    frame #5995: 0x0000000100994e71 NIO`specialized SelectableEventLoop.handleEvent<A>(_:channel:) [inlined] generic specialization <NIO.Socket> of protocol witness for NIO.SelectableChannel.reset() -> () in conformance NIO.BaseSocketChannel<A> : NIO.SelectableChannel in NIO at <compiler-generated>:912 [opt]
    frame #5996: 0x0000000100994e71 NIO`specialized SelectableEventLoop.handleEvent<A>(ev=<unavailable>, channel=<unavailable>) at EventLoop.swift:771 [opt]
    frame #5997: 0x00000001009e49ad NIO`specialized Selector.whenReady(strategy:_:) [inlined] generic specialization <NIO.SocketChannel> of NIO.SelectableEventLoop.(handleEvent in _D5D78C61B22284700B9BD1ACFBC25157)<A where A: NIO.SelectableChannel>(_: NIO.SelectorEventSet, channel: A) -> () at <compiler-generated>:0 [opt]
    frame #5998: 0x00000001009e499a NIO`specialized Selector.whenReady(strategy:_:) at EventLoop.swift:833 [opt]
    frame #5999: 0x00000001009e499a NIO`specialized Selector.whenReady(strategy:_:) [inlined] reabstraction thunk helper from @callee_guaranteed (@guaranteed NIO.SelectorEvent<NIO.NIORegistration>) -> (@error @owned Swift.Error) to @escaping @callee_guaranteed (@in_guaranteed NIO.SelectorEvent<NIO.NIORegistration>) -> (@error @owned Swift.Error) at <compiler-generated>:0 [opt]
    frame #6000: 0x00000001009e499a NIO`specialized Selector.whenReady(strategy=<unavailable>, self=<unavailable>) at Selector.swift:588 [opt]
    frame #6001: 0x000000010098b507 NIO`closure #1 in SelectableEventLoop.run(self=0x0000000105a23220, nextReadyTask=0x00007fff7632eaa0) at EventLoop.swift:828:30 [opt]
    frame #6002: 0x0000000100992161 NIO`partial apply for closure #1 in SelectableEventLoop.run() at <compiler-generated>:0 [opt]
    frame #6003: 0x000000010092c05c NIO`thunk for @callee_guaranteed () -> (@error @owned Error) at <compiler-generated>:0 [opt]
    frame #6004: 0x00000001009921a1 NIO`partial apply for thunk for @callee_guaranteed () -> (@error @owned Error) at <compiler-generated>:0 [opt]
    frame #6005: 0x00000001009921e1 NIO`partial apply for specialized  [inlined] generic specialization <()> of closure #1 (execute=<unavailable>) throws -> A in NIO.(withAutoReleasePool in _D5D78C61B22284700B9BD1ACFBC25157)<A>(() throws -> A) throws -> A at EventLoop.swift:587:13 [opt]
    frame #6006: 0x00000001009921de NIO`partial apply for specialized  at <compiler-generated>:0 [opt]
    frame #6007: 0x00007fff76624f4f libswiftObjectiveC.dylib`ObjectiveC.autoreleasepool<A>(invoking: () throws -> A) throws -> A + 47
    frame #6008: 0x000000010098af14 NIO`SelectableEventLoop.run() [inlined] generic specialization <()> of NIO.(execute=0x0000000000000000)<A>(() throws -> A) throws -> A at EventLoop.swift:586:16 [opt]
    frame #6009: 0x000000010098aefd NIO`SelectableEventLoop.run(self=0x0000000105a23220) at EventLoop.swift:827 [opt]
    frame #6010: 0x000000010098cb6e NIO`closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(t=<unavailable>, initializer=<unavailable>, lock=(mutex = 0x000000010134a8a0), _loop=0x0000000105a23220, loopUpAndRunningGroup=0x0000000101326d90) at EventLoop.swift:1035:23 [opt]
    frame #6011: 0x0000000100999309 NIO`partial apply for closure #1 in static MultiThreadedEventLoopGroup.setupThreadAndEventLoop(name:initializer:) at <compiler-generated>:0 [opt]
    frame #6012: 0x0000000100a01e24 NIO`partial apply for thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () [inlined] reabstraction thunk helper from @escaping @callee_guaranteed (@guaranteed NIO.NIOThread) -> () to @escaping @callee_guaranteed (@in_guaranteed NIO.NIOThread) -> (@out ()) at <compiler-generated>:0 [opt]
    frame #6013: 0x0000000100a01e1e NIO`partial apply for thunk for @escaping @callee_guaranteed (@guaranteed NIOThread) -> () at <compiler-generated>:0 [opt]
    frame #6014: 0x0000000100a01783 NIO`closure #1 in static NIOThread.spawnAndRun(p=(_rawValue = 0x000000010133fe50 -> 0x00007fffa9866f98 libswiftCore.dylib`InitialAllocationPool + 29472)) at Thread.swift:104:13 [opt]
    frame #6015: 0x0000000100a017c9 NIO`@objc closure #1 in static NIOThread.spawnAndRun(name:body:) at <compiler-generated>:0 [opt]
    frame #6016: 0x00000001010eadc3 libsystem_pthread.dylib`_pthread_body + 126
    frame #6017: 0x00000001010ede8d libsystem_pthread.dylib`_pthread_start + 66
    frame #6018: 0x00000001010e9e11 libsystem_pthread.dylib`thread_start + 13
(lldb) 

Not sure if this is useful. I will keep this Xcode run around for a while. If you would like me to inspect something, let me know. - although this is release mode...

[Lukasa]

This seems to be an infinite, or at least very long, recursion. That suggests there is a promise that is failing promises infinitely in a loop.

One possible source of this is redirection: it seems like async-http-client doesn't place a maximum redirect limit, and very long redirect chains can potentially cause nearly infinite callback chains as well. That looks likely to be a source of bugs: it seems like the redirect code probably needs a way to "take over" the task for an existing piece of work, or alternatively to bound the maximum allowed number of redirects to something sensible.

[timprepscius]

Umm here is the output directly before the crash.
It is scraping in parallel, so there is no guarantee that the error directly before the crash, applies to the error for the crash

NIOConnectionError(host: "192.99.191.232", port: 8080, dnsAError: nil, dnsAAAAError: nil, connectionErrors: [NIO.SingleConnectionFailure(target: [IPv4]192.99.191.232/192.99.191.232:8080, error: connection reset (error set): Connection refused (errno: 61))])
JSONURLSessionManagerNIO: cycling
JSONURLTask: web read of https://graphs2.somecompany.com/currencies/diruna/1509148800000/1509667200000/
statusCodeNot200
JSONURLSessionManagerNIO: cycling
2019-08-09 00:26:47.141924-0400 AltCoin-Sim[29354:691796] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C1.1:2][0x101306d70] get output frames failed, state 8196
2019-08-09 00:26:47.141979-0400 AltCoin-Sim[29354:691796] [BoringSSL] nw_protocol_boringssl_get_output_frames(1301) [C1.1:2][0x101306d70] get output frames failed, state 8196
2019-08-09 00:26:47.142338-0400 AltCoin-Sim[29354:691796] TIC Read Status [1:0x0]: 1:57
2019-08-09 00:26:47.142365-0400 AltCoin-Sim[29354:691796] TIC Read Status [1:0x0]: 1:57
(lldb) 

each time it cycles, it is switching to a new proxy. a proxy is used only by one thread.

[Lukasa]

I don't think this has anything much to do with it: redirects are handled without your input, so your logs are unlikely to see the problem.

[timprepscius]

Looking through your stack trace it seems like

        case (.some(let first), .some(let others)):
            var pending = [first]+others
            while pending.count > 0 {
                let list = pending
                pending = []
                for f in list {
 ***                   let next = f()
                    pending.append(contentsOf: next._allCallbacks())
                }
            }
        }
    }

calls:

    @inlinable
    public func whenComplete(_ callback: @escaping (Result<Value, Error>) -> Void) {
        self._whenComplete {
***            callback(self._value!)
            return CallbackList()
        }
    }

going to

    @inlinable
    public func cascade(to promise: EventLoopPromise<Value>?) {
        guard let promise = promise else { return }
        self.whenComplete { result in
            switch result {
            case let .success(value): promise.succeed(value)
***            case let .failure(error): promise.fail(error)
            }
        }
    }

going to:

    @inlinable
    public func fail(_ error: Error) {
***        self._resolve(value: .failure(error))
    }

going to:

    @inlinable
    internal func _resolve(value: Result<Value, Error>) {
        if self.futureResult.eventLoop.inEventLoop {
***            self._setValue(value: value)._run()
        } else {
            self.futureResult.eventLoop.execute {
                self._setValue(value: value)._run()
            }
        }
    }

which goes back to:

        case (.some(let first), .some(let others)):
            var pending = [first]+others
            while pending.count > 0 {
                let list = pending
                pending = []
                for f in list {
***                    let next = f()
                    pending.append(contentsOf: next._allCallbacks())
                }
            }
        }
    }

[timprepscius]

Ok will be back later, let me know if you want any contents to any variable.

[Lukasa]

That's correct, but that can only happen if in the callback fired as part of f we return a new callback list. Realistically, so far as I can see, this can only happen with a very long callback chain: I have not been able to trigger this myself by trying to cascade promises into previously succeeded promises.

[timprepscius]

Interesting - this might also be the "getting stuck" problem (just I ran it long enough to see the eventual crash) Neat!!!

…

On Fri, Aug 9, 2019 at 8:38 AM Cory Benfield ***@***.***> wrote: That's correct, but that can only happen if in the callback fired as part of f we return a new callback list. Realistically, so far as I can see, this can only happen with a very long callback chain: I have not been able to trigger this myself by trying to cascade promises into previously succeeded promises. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#76?email_source=notifications&email_token=AAZQXZVHSODHSGE6AIBICWTQDVQMXA5CNFSM4IKEPWK2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD36RL7A#issuecomment-519902716>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAZQXZSQUSOIZD3IVZ4UD33QDVQMXANCNFSM4IKEPWKQ> .

[timprepscius]

If you put in a fix for the redirect, could you comment on this thread, so I can test. (I would also like to get rid of the Foundation http classes, because I can't get rid of that pop up, and then it blocks the program)