PSReadline cannot keep up with keyboard input; leaves raw escape codes on screen

[cspotcode]

Environment data

PS version: 6.0.2
PSReadline version: 2.0.0-beta2
os: Linux ABradley-T460s 4.4.0-17134-Microsoft #112-Microsoft Thu Jun 07 22:57:00 PST 2018 x86_64 x86_64 x86_64 GNU/Linux
PS file version:
/mnt/c/Users/abradley

Steps to reproduce or exception report

Open PSReadline within pwsh on WSL Ubuntu. I pressed Win+R, typed "ubuntu.exe -c pwsh" and hit Enter.

Enter a long line of input. Position the cursor at the beginning of the input. Hold the "Delete" key. The input will rapidly disappear one character at a time (good) but every so often, the "Del" escape code will be written to the display. ^[[3~ (bad)

It's as if PSReadline can't keep up when keystrokes are being input so quickly, so some of them "escape" onto the screen, and the escape code is displayed raw. Which doesn't make much sense to me; I assume all input should go into some sort of queue and never be rendered visually till PSReadline processes it. Is that not what's happening?

Possibly related to my comments on #420: #420 (comment)

[cspotcode]

I noticed another quirk. When using -EditMode Vi, typing [3 prompts for "digit-argument: 3" Except, when the Del escape code appears on screen, it contains [3, so 2 things happen:
digit-argument: 3 flashes at the bottom of the screen, and if our prompt is already at the bottom of the console, the whole thing scrolls one line up to accommodate the digit-argument prompt.

[cspotcode]

Could this be a race condition where PSReadline is actively consuming input while the OS / console is actively pushing new inputs?

For example, if I hit the Del key twice, this should go into the queue: ^[[3~^[[3~

But if something goes wrong (this is a shot in the dark) then maybe the second escape code gets inserted within the first one? Or they get interleaved in some other way? ^[^[[3~[3~

EDIT: or is it related to this? https://github.com/lzybkr/PSReadLine/blob/3229956a1973ffe65c24dbbe9ba421e62bcbf0b3/PSReadLine/Cmdlets.cs#L124-L128

[lzybkr]

It's possible that #420 is the root cause or at least having some impact, I honestly don't know.

The VT escape processing happens in .Net, not in PSReadLine. You can start here to see the implementation.

The code you've found is only relevant to running Windows PowerShell or PowerShell Core on Windows, and even then, you have to go out of your way to hit that code by setting an environment variable. That code exists so Windows PowerShell can work on WSL so people can use it from screen or tmux, and AFAIK, only the contributor is doing that right now. At any rate, the VT processing code in PSReadLine is definitely not causing problems on Linux.

[cspotcode]

EDIT: I talked to @SeeminglyScience on Slack, who pointed me to where you have already discussed this: #626
537f5f2#r165221488

Thanks for the info; I'm very new to all of this. But I think I figured it out! PSReadline needs to turn off tty echo while it's accepting user input. As a proof of concept, running stty -echo (unix utility) makes the problem disappear. The proper solution is for PSReadline to disable echo before reading a line and re-enable it immediately after.

DotNet corefx does this via tcsetattr before every call to ReadKey and immediately restores it afterward. This means when the key-reading thread is between calls to ReadKey, it's possible for keyboard input and escape sequences to be rendered in the console.

https://github.com/dotnet/corefx/blob/a10890f4ffe0fadf090c922578ba0e606ebdd16c/src/Native/Unix/System.Native/pal_console.c#L111-L146

What I don't yet know is if corefx exposes an API to set terminal echo on and off. I don't think it does, so my next question is how hard it'll be for PSReadline to ship with some native code to do it. (you could call the Unix stty utility but that's hardly performant)

Here's a hack to solve the problem, but calling an external binary noticably slows the prompt and doesn't handle all situations:

# In $PROFILE
function prompt {
    # Disable echo before prompt
    stty -echo
    return "$PWD > "
}
# Re-enable echo after prompt
Set-PSReadLineKeyHandler -Key Enter -ScriptBlock {stty echo ; [Microsoft.PowerShell.PSConsoleReadLine]::AcceptLine()}

---

Reproductions:

# In pwsh on Linux / WSL
$a = [console]::ReadKey($true) ; sleep 1 | out-null ; $b = [console]::readkey($true) ; $a ; $b

Tap the Delete key twice quickly and see the following:

^[[3~
KeyChar    Key Modifiers
-------    --- ---------
        Delete         0
        Delete         0

Both keypresses are read and 2 instances of ConsoleKeyInfo returned. The first keypress is hidden but the second displays ^[[3~ since it happens before the second call to ReadKey.

Bash example with echo disabled; escape sequences are not rendered:

stty -echo
echo waiting for keypress ; read -n 3 -s
echo waiting 5 seconds ; sleep 5
echo press key again ; read -n 3 -s
echo done
stty echo

[lzybkr]

I like the stty -echo idea, and that's a super easy fix if you'd like to try it out.

Replace the body of PSReadLine.psm1 with

if (Get-Command stty) {
    function PSConsoleHostReadLine
    {
        stty -echo
        Microsoft.PowerShell.Core\Set-StrictMode -Off
        [Microsoft.PowerShell.PSConsoleReadLine]::ReadLine($host.Runspace, $ExecutionContext)
        stty echo
    }
}
else {
    function PSConsoleHostReadLine
    {
        Microsoft.PowerShell.Core\Set-StrictMode -Off
        [Microsoft.PowerShell.PSConsoleReadLine]::ReadLine($host.Runspace, $ExecutionContext)
    }
}

Actually, you can just run this at your console to try it out.

[SeeminglyScience]

@cspotcode If it's just the escape characters making it through that is giving you issues, try out a preview build of PowerShell Core 6.1.

There's some fixes for the handling of escape characters in .NET Core 2.1 which ships with 6.1. That might solve your issues completely.

[cspotcode]

@lzybkr I wrote an in-process implementation of stty -echo and stty echo. It calls stty echo and stty -echo once to detect which bit in termios is being toggled. Thereafter it can call the native tcgetattr and tcsetattr calls, which is fast enough to call for every prompt. It's definitely a hack, but it works for me. I'm curious to hear your thoughts if you have a second. https://github.com/cspotcode/ps-tty-echo

@SeeminglyScience thanks, I'm giving it a try. It seems that when calling ReadKey in a while loop and holding down the "Del" key, the stream of keys being read is not exclusively "Del". Sometimes it misinterprets one as its constituent parts. I'll see a stream of "Escape", "3", "~" (it drops the "["??) or more rarely "Escape", "[", "3", "~". I can even reproduce this behavior by piping the raw ANSI sequence into a simple dotnet executable in a loop. It seems that ANSI sequences need to be written with very good timing or else the receiving process might split them up.

tl;dr I think there are 2 separate problems:

1. tty echo causing ANSI escapes to appear. Fixed by my tcsetattr hack above.
2. .NET (rarely) splitting up a single ANSI escape sequence into its constituent parts, as 3 or 4 different keystrokes.

[daxian-dbw]

As @cspotcode pointed out at https://github.com/dotnet/corefx/issues/37948#issuecomment-499306816, the echo issue has been fixed in .NET Core 3.0, quoted from the commit message of the specific change:

This changes to echoing while there are interactive applications and not echoing
when there are none.

This means we no longer need to toggle echo off/on for each Console.Read.
And the terminal will no longer echo when there is no Console.Read going on.