Copy/pasting: multiline in a single line prompt

[gpotter2]

Could we have an option so that prompt only accepts a single line, even when pasting some text ?

Currently, even when multiline is set to False, you can paste multi-line text in it.

Code: Here's the code i am using:

PROMPT_TOOLKIT_HISTORY = prompt_toolkit.history.FileHistory(conf.histfile)
# ScapyCompleter is a quite basic completer, returning all matching content as a shell would do
_completer = ScapyCompleter()
from prompt_toolkit.token import Token
prompts_style = prompt_toolkit.styles.style_from_dict({
    Token.Prompt: "#2E64FE",
})
def readLineScapy(prompt):
    result = ""
    end = False
    while not end :
        if not end and result != "":
            line = prompt_toolkit.prompt(u"... ", completer=_completer, history=PROMPT_TOOLKIT_HISTORY)
        else:
            line = prompt_toolkit.prompt(u">>> ", completer=_completer, style=prompts_style, history=PROMPT_TOOLKIT_HISTORY)
        if line.strip()[-1:] in [":", "(", ",", "\\"]:
            end = False
        elif result == "":
            end = True
        if line.strip() == "":
            end = True
        result = result + "\n" + line
    result = result.replace("\\\n", "")
    return str(result)
# Run console with python_toolkit
code.interact(banner="hello, this is a great shell", readfunc=readLineScapy)

The goal of this code is to emulate a shell. In that way, we need to print "..." or ">>>" on each line. readlinescapy is using prompt_toolkit.prompt to get the same input as in a shell...

When one copy multi-line, the prompt function takes all the lines, so ... is not printed.

[jonathanslenders]

Hi @gpotter2,

At this point, the value of multiline only affects what the enter key does. If True, then pressing enter will insert a newline and meta+enter is required to accept the input. Otherwise, enter will accept the input directly.

Pasting text is however not the same as pressing enter. A paste is recognised and ends up in a different key binding (bracketed paste is enabled). If the pasted text contains an newline, then this turns the input into a multiline buffer. It's different from how readline works (which doesn't support bracketed paste), but it's considered a good thing, because this behaviour allows the user to review any pasted input, before actually executing it.

Prompt_toolkit can render >>> and ...``` for every next line automatically. Check the get_multiline_inputexample. If you need multiline input, it's better to have prompt_toolkit handle it, rather than doing a while loop over multipleprompt`` calls. That way, the user can use the up/down arrows to edit the input.

Let me know what you think.
If there's a big limitation or shortcoming in the current design, I'd like to make sure that the 2.0 version will do the right thing.

[gpotter2]

Hi @jonathanslenders, thanks for answering

We cannot use the multiline python-prompt-toolkit implementation because of several limitations:

• We want to emulate console: that means that pressing "Enter" twice should return. Not sure that this is possible in python-prompt-toolkit
• Single lines should be detected as so. This means that typing for instance a=1 should not trigger the multiline support... What could fix this is to add a parameter use_multiline_if, that would be call when pressing for the first time "Enter". Something so we could perform an API call like:

def continuation_tokens(cli, width):
    return [(Token, '... ')]
# Would be triggered only once, when pressing enter at the end of first line
def is_multi_line(line):
    return line.strip() and not line.strip()[-1:] in [":", "(", "[", "\\"]
prompt_toolkit.prompt(u">>> ", get_continuation_tokens=continuation_tokens, use_multiline_if=is_multi_line)

Apart from those, python-prompt-toolkit is great !!

[asmeurer]

You can add a custom key binding for BracketedPaste (this is the default implementation).

Assumedly you would want a multiline paste to act like entering multiple commands. You could split on newline and force it to execute each line separately. I don't know if it's the best way to do it, but I think it's possible using a PipeInput.

Single lines should be detected as so. This means that typing for instance a=1 should not trigger the multiline support... What could fix this is to add a parameter use_multiline_if, that would be call when pressing for the first time "Enter". Something so we could perform an API call like:

multiline can be passed a Condition. Look at how it is done in the python-prompt-toolkit source. I personally recommend doing real tokenization if you want to get all the cases with ( and " and so on right.

[gpotter2]

Thanks for your help,

I'll create a custom class extending Filter as multiline parameter, and use tokenizing. That should be it for multi lines :)

I'll come back if I have any problems :)

[asmeurer]

This is the function I used to detect if a block of Python code is "multiline" or not, using the tokenize module. Feel free to reuse that code if it helps you (there are other useful functions in that file too). This also goes for ptpython (this is a more robust method than what is used there).

[gpotter2]

@asmeurer Thanks a lot for the tools !

[gpotter2]

@asmeurer Hello, thanks for your help.
Although, i have tried passing a condition to multiline as so:
multiline=Condition(lambda cli: is_multiline_python(cli.current_buffer.current_line))

which gave me:

Traceback (most recent call last):
  File "C:\Python27\lib\runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "C:\Python27\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "Z:\Coding\github\scapy\scapy\__init__.py", line 93, in <module>
    interact()
  File "scapy\main.py", line 418, in interact
    local=session, readfunc=conf.readfunc)
  File "C:\Python27\lib\code.py", line 306, in interact
    console.interact(banner)
  File "C:\Python27\lib\code.py", line 234, in interact
    line = self.raw_input(prompt)
  File "scapy\main.py", line 412, in readLineScapy
    completer=_completer, multiline=Condition(lambda cli: is_multiline_python(cli.current_buffer.current_line)))
  File "C:\Python27\lib\site-packages\prompt_toolkit\shortcuts.py", line 541, in prompt
    application = create_prompt_application(message, **kwargs)
  File "C:\Python27\lib\site-packages\prompt_toolkit\shortcuts.py", line 465, in create_prompt_application
    multiline = to_simple_filter(multiline)
  File "C:\Python27\lib\site-packages\prompt_toolkit\filters\utils.py", line 20, in to_simple_filter
    raise TypeError('Expecting a bool or a SimpleFilter instance. Got %r' % bool_or_filter)
TypeError: Expecting a bool or a SimpleFilter instance. Got Condition(<function <lambda> at 0x000000000B0C1828>)

The doc isn't helpful:

is_multiline – SimpleFilter to indicate whether we should consider this buffer a multiline input. If so, key bindings can decide to insert newlines when pressing [Enter]. (Instead of accepting the input.)

Sadly the doc about SimpleFilter is very much like the void:

class prompt_toolkit.filters.SimpleFilter
Abstract base class for filters that don’t accept any arguments.

Any ideas ?

[gpotter2]

I've found a way of avoding this mess. Thanks for your help.

[asmeurer]

This may be of interest here, I recently wanted to implement some similar behavior. Basically, I want to have it so that if I copy some text from a Python example, like say

>>> x = np.array([[1, 2, 3], [4, 5, 6]], np.int32)
>>> type(x)
<type 'numpy.ndarray'>
>>> x.shape
(2, 3)
>>> x.dtype
dtype('int32')

(taken from the numpy docs), that it automatically strips the prompts and outputs. This is quite easy to do by setting a key for Keys.BracketedPaste and preprocessing the input with a regular expression (I also use tokenization to only do this when not pasting inside a string).

But when I do this, it pastes everything as a single multiline block, like

x = np.array([[1, 2, 3], [4, 5, 6]], np.int32)
type(x)
x.shape
x.dtype

meaning I only see the output of the last line. I really want it to paste an execute each line, as it would in a readline-style prompt (actually readline 6, since readline 7 also supports bracketed paste).

At first, I tried to create a custom Input subclass that mixed stdin and a pipe, but this seems to be impossible to make work. The internals of prompt-toolkit use the fileno (the read method isn't even used), meaning if you want to do something like this, it has to be supported at the os level. You can sort-of do this with select, but it requires returning a different fileno based on if the pipe has data, and this confuses prompt-toolkit's internals. It would be really nice if prompt-toolkit worked on file-like objects instead of file numbers, but from what I've seen of the eventloop code, this could require some refactoring.

Next, I tried to pass two file numbers to the eventloop, as it apparently supports this. But I was unable to figure out how to make the input from the pipe actually get read. Is the handler supposed to do this? Can it be done without access to the data internal local to the run method? Maybe this is possible, but I couldn't figure it out, and I wasn't keen on subclassing EventLoop, which it was seeming like I would have to do.

Finally, I figured out a much more elegant solution. In my main loop, where I create a CommandLineInterface for each input, I pass in the input argument. I have a queue, like

from collections import deque
from prompt_toolkit.input import PipeInput

CMD_QUEUE = deque()
def main_loop():
    ...
    while True:
        if CMD_QUEUE:
            _input = PipeInput()
            _input.send_text(CMD_QUEUE.popleft() + '\n')
        else:
            _input = None # The default stdin input

        cli = CommandLineInterface(..., input=_input)
        result = cli.run()

        # Code to execute the result
        ...

Then in the bracketed paste handler, I have some code that splits up the inputs in the way I want, and finally does:

# inputs is a list of split up inputs
event.current_buffer.insert_text(inputs[0])

for text in inputs[1:]:
    # TODO: Send last chunk as bracketed paste, so it can be edited
    CMD_QUEUE.append(text)
if CMD_QUEUE:
    accept_line(event)

(I am still figuring out the TODO part).

This works well, except prompt toolkit has a lot of problems with pipe inputs, where the prompts get messed up and CPR codes get printed (#456), so you have to be willing to deal with that.

I hope this helps. It should be possible to use this idea to do what you were wanting to do (if you indeed still want to do that).

[jonathanslenders]

Hi @asmeurer,

Could you copy your comment in a new issue? That way we won't forget about this.
I still did not have time to think about this or look for a solution, but I'd like to keep it in mind.

[asmeurer]

My main issue on the prompt-toolkit side is with the Input classes. If it's possible, it would be nice if they didn't require a fileno, but it isn't completely clear to me from the code if it is possible. Also it seems they should just be file-like objects. I can open an issue about that.

For the rest, it only seems like something that could go in prompt-toolkit if you ever want to add some abstractions for REPL loops (an abstraction around the common while True: prompt() loop). For me personally, I already stopped using the prompt-toolkit provided abstractions prompt_toolkit.shortcuts.prompt and prompt_toolkit.shortcuts.create_application a while ago, because they didn't provide enough customization for what I wanted. I create Buffer (actually a Buffer subclass), Application, and CommandLineInterface objects manually (I'm still using create_prompt_layout, but that may change change too, as I more and more customize things to my own tastes).

But for users who are less picky about customizing and are fine with prompt-toolkit defaults, these shortcut functions are nice. And they're great for starting because you can get up and running very quickly with a basic prompt application, and then build on top of them once you get the hang of the library.

So maybe a shortcut function like

def repl(execute_callback, message='', **kwargs):
    while True:
        try:
            res = prompt(message, **kwargs)
        except KeyboardInterrupt:
            print("KeyboardInterrupt", file=sys.stderr)
            continue
        except (EOFError, SystemExit):
            break
        execute_callback(res) # Perhaps the callback should also be passed the cli instance

in shortcuts.py could be useful. And then you could build on that to do the queueing behavior, and possibly other behaviors too (e.g., I keep IPython-style prompt numbers in my loop).