MuxedStream for STDIN does not close itself properly

[admilazz]

When a MuxedStream representing STDIN is closed it should send a message indicating this to the remote side. As a simple example, if you execute /bin/cat on a pod, send some data over STDIN, and dispose the STDIN stream, the command will never complete because the closure of the stream is not communicated to the server, so cat waits forever for STDIN to be closed.

[admilazz]

It should suffice to execute this:

await socket.SendAsync(new byte[] { (byte)ChannelIndex.StdIn }, WebSocketMessageType.Binary, true, cancelToken)
await socket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, null, cancelToken)

when the STDIN stream is closed. EDIT: Turns out that's not quite sufficient. See below.

[admilazz]

Turns out that's not quite sufficient, since it intermittently truncates the data in STDOUT/STDERR. I think the RunLoop method in StreamDemuxer.cs must also be fixed to wait for a full closure, or else I need to figure out the correct protocol to close STDIN without closing the whole socket...

[admilazz]

In the SPDY protocol (which kubectl exec uses), the closing of STDIN is signaled by a zero-byte data frame with the FIN bit set, and this I've verified from a hex dump of the network traffic.

Presumably you do the same thing for web sockets. I've tried sending an empty frame with the finish flag...
await socket.SendAsync(new byte[] { (byte)ChannelIndex.StdIn }, WebSocketMessageType.Binary, true, cancelToken)
but it doesn't seem to close the stream. (Or if it does close the stream, the client remains blocked in webSocket.ReceiveAsync.)

I've tried doing that and then following it up with a second empty frame with the finish flag, thinking that maybe it's looking for an empty message and not just a frame. No dice.

I've also tried sending a zero-byte message, without the channel index.
await socket.SendAsync(new byte[0], WebSocketMessageType.Binary, true, cancelToken)
Same thing. StreamDemuxer remains blocked on ReceiveAsync.

I've tried doing all three together with no luck.

The only thing that consistently ends the command is to call CloseOutputAsync:
await socket.CloseOutputAsync(WebSocketCloseStatus.NormalClosure, null, cancelToken)
i.e. to do a "half close" of the connection. But the server doesn't seem to interpret that as "finish sending everything and then close". It seems to interpret it as "close ASAP", which may truncate the data being sent back. Or perhaps it's the client in StreamDemuxer that's not doing things right.

It's all made so difficult by the lack of any Kubernetes websocket protocol documentation that I could find.

[admilazz]

After trawling through the Kubernetes source code, I've come to suspect that there is no protocol for closing STDIN over web sockets (although there is over SPDY). If so, that is probably a bug in Kubernetes, but the C# client still needs some provision for communicating the closure of STDIN eventually.

Perhaps it should switch to SPDY until this problem is fixed.

[admilazz]

I filed a bug against Kubernetes for the protocol problem.

I also implemented exec over SPDY, and it solves this problem nicely. I'd be glad to contribute it to the C# client, but there is one difficulty: the SPDY algorithm requires zlib compression, and I haven't found a managed implementation capable of satisfying the particular requirements of SPDY (though I probably didn't look at all of them). In my implementation I simply p/invoke libz, which is part of the Linux Standard Base (LSB) and reliably available on basically any GNU/Linux system and probably any other flavor of *nix system, but not likely to be available on Windows. I think it's possible to bundle a native zlib DLL in the Nuget package, but it may then be necessary to deal with 32-bit and 64-bit versions and I don't know if you guys want to get into that. Thoughts?

[tg123]

nuget packages support multiple platforms and can ship with native dlls by adding runtimes folder

BUT, are you going to create an SPDY httpclient?

I remember dotnet supports http2

[admilazz]

I created a SPDY exec implementation and integrated it into the Kubernetes client locally. I'd be glad to contribute it but I'm not too keen on figuring out the cross-platform issues around zlib, especially obtaining 32-bit and 64-bit machines and testing on them... (I don't know if we care about 32-bit.)

.NET may support HTTP2, but Kubernetes doesn't use it for exec yet.

[tg123]

did you have a look at https://www.nuget.org/packages?q=zlib
I believe there is pure netstandard version

[admilazz]

I did that search. The top result had no documentation that I could see. (The documentation link goes to http://cheeso.members.winisp.net/DotNetZipHelp/frames.htm, which doesn't exist and looks like a pretty amateur URL for a project to be using anyway.) So I skipped past it and looked at some of the others zlib implementations for .NET, none of which did what I needed.

That said, I just took the top result and decompiled it, and in fact it looks very promising. It seems like maybe a straight recompilation of the core zlib code with all the advanced features. So it might work after all!

… unfortunately, it has a very wonky interface, but if it works then it works...

[admilazz]

Alright, I got it working with Ionic.Zlib (which seems like a straight port of the real zlib). Now I just have to figure out the legal mumbojumbo needed to publish the source code...

[brendandburns]

Cool, we'd be happy to take a PR with this.

[admilazz]

Well, I implemented it but it doesn't work in the net452 target, because the HTTP response stream there is read-only, and furthermore is empty on an upgraded connection. It seems the HttpClient in .NET Framework 4.x simply isn't meant for this. (I see it has some special logic for connections via the web socket client, but it doesn't seem possible to trigger that path, and I'm not sure it'd work anyway.)

So currently it's wrapped in #if !NET452.

Probably the only way to make it work is with some reflection-based hack on the net452 target. Still want it?

[admilazz]

It works with the reflection hack to get the underlying network stream in the net452 target.