AssertionError cursor_position is None

[jaraco]

I use xonsh as my shell with prompt-toolkit 3.0.4. Occasionally, I'll be typing in the prompt and get this traceback:

Unhandled exception in event loop:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/eventloop/utils.py", line 72, in schedule
    func()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 456, in redraw
    self._redraw()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 523, in _redraw
    self.context.run(run_in_context)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 509, in run_in_context
    self.renderer.render(self, self.layout)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/renderer.py", line 568, in render
    layout.container.preferred_height(size.columns, size.rows).preferred,
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 792, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 2595, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1620, in preferred_height
    return self._merge_dimensions(
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1648, in _merge_dimensions
    preferred = get_preferred()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1613, in preferred_content_height
    return self.content.preferred_height(
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 639, in preferred_height
    content = self.create_content(width, height=1)  # Pass a dummy '1' as height.
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 761, in create_content
    document = buffer.document
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 524, in document
    return self._document_cache[
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/cache.py", line 98, in __missing__
    result = self.get_value(*key)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/document.py", line 98, in __init__
    assert cursor_position is None or cursor_position <= len(text), AssertionError(

Exception cursor_position=9, len_text=4
Press ENTER to continue...                                                                                                                                                
~ $ cd ~/p/che

Sometimes, I hit enter and the error repeats. Other times, I hit Enter and can continue.

I've just updated to prompt_toolkit==3.0.5. I'll report back here if the issue repeats.

[jonathanslenders]

My first idea is that this is possibly a broken key binding that moves the cursor beyond the text length.
If we have a reliable way to reproduce it, that would be great.

This traceback here is what happens during the rendering of the output to the screen, the actual error is a buffer manipulation some time before this.

[jaraco]

I usually notice the issue when I've been typing into the shell while it's starting up. I suspect, though can't confirm, that I'm both entering and deleting characters. I tried replicating the issue by entering and deleting lots of characters during startup, but thusfar has been unsuccessful in triggering the behavior. I also haven't experienced the issue since, so maybe 3.0.5 addresses the issue. I'll do my best to capture more precisely the conditions that cause the issue next time it happens.

[hroemer]

This issue pops up quite often on my end when opening alacritty with xonsh/xonsh#3522 as default shell.

[jaraco]

The issue just happened again. I started up a new shell and was typing into the shell as it was starting up:

~ $ cd ~/p/pypa/setuptools

Unhandled exception in event loop:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/eventloop/utils.py", line 72, in schedule
    func()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 456, in redraw
    self._redraw()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 523, in _redraw
    self.context.run(run_in_context)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 509, in run_in_context
    self.renderer.render(self, self.layout)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/renderer.py", line 617, in render
    layout.container.preferred_height(size.columns, size.rows).preferred,
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 792, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 2595, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1620, in preferred_height
    return self._merge_dimensions(
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1648, in _merge_dimensions
    preferred = get_preferred()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1613, in preferred_content_height
    return self.content.preferred_height(
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 639, in preferred_height
    content = self.create_content(width, height=1)  # Pass a dummy '1' as height.
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 761, in create_content
    document = buffer.document
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 524, in document
    return self._document_cache[
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/cache.py", line 98, in __missing__
    result = self.get_value(*key)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/document.py", line 98, in __init__
    assert cursor_position is None or cursor_position <= len(text), AssertionError(

Exception cursor_position=8, len_text=7
Press ENTER to continue...

...

Hitting ENTER emits the exact same error.

I'd say this happens about 1/100 times. I'm unable to replicate it reliably.

As you can see, I'd entered 22 characters plus carriage return (maybe, I'm not certain about the CR), but the buffer manipulation seems to have gone awry before that. I can't remember if I hit backspace. For this command, I give a <20% chance I hit backspace while typing it.

[jaraco]

Today it happened again. Again, I'd started up a clean terminal and started typing ls sudoers^h^h^h^h^h^h^h/etc/s (or thereabouts, when the error occurred):

Unhandled exception in event loop:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 653, in read_from_input
    self.key_processor.process_keys()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_processor.py", line 274, in process_keys
    self._process_coroutine.send(key_press)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_processor.py", line 186, in _process
    self._call_handler(matches[-1], key_sequence=buffer[:])
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_processor.py", line 329, in _call_handler
    handler.call(event)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_bindings.py", line 101, in call
    self.handler(event)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/bindings/named_commands.py", line 270, in self_insert
    event.current_buffer.insert_text(event.data * event.arg)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 1209, in insert_text
    self.document = Document(text, cpos)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/document.py", line 98, in __init__
    assert cursor_position is None or cursor_position <= len(text), AssertionError(

Exception cursor_position=7, len_text=5
Press ENTER to continue...
~ $ ls /etc/ssh

This time, as you can see, hitting ENTER dismissed the error and I could see my partially-entered command with some autocomplete present:

Here, neither the entered text len('ls /et') == 6 nor the cursor position (6 if the empty prompt is position 0) appear to correspond to the values in the exception.

If you can think of a way that I could add some debugging to prompt_toolkit or xonsh to help capture the issue, I'm happy to do that.

[hroemer]

I can reproduce it with approx. every 3rd try by launching a terminal and quickly search for a command in the history (CTRL + R). The weird thing is that I now get 3 different exceptions here:

cd .local/lib/python3.8/site-packages/compose/                                                        
Exception in thread Thread-11:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 932, in _bootstrap_inner
Exception in threading.excepthook:
Traceback (most recent call last):
  File "/usr/lib/python3.8/threading.py", line 1202, in invoke_excepthook
    hook(args)
  File "/usr/lib/python3.8/site-packages/xonsh/__amalgam__.py", line 17609, in flush
    self.std.flush()
AttributeError: 'NoneType' object has no attribute 'flush'

or

Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/xonsh/__amalgam__.py", line 24210, in main
    return main_xonsh(args)
  File "/usr/lib/python3.8/site-packages/xonsh/__amalgam__.py", line 24239, in main_xonsh
    shell.shell.cmdloop()
  File "/usr/lib/python3.8/site-packages/xonsh/ptk_shell/shell.py", line 197, in cmdloop
    line = self.singleline(auto_suggest=auto_suggest)
  File "/usr/lib/python3.8/site-packages/xonsh/ptk_shell/shell.py", line 166, in singleline
    line = self.prompter.prompt(**prompt_args)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/shortcuts/prompt.py", line 994, in prompt
    return self.app.run(set_exception_handler=set_exception_handler)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 811, in run
    return loop.run_until_complete(
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 778, in run_async
    return await _run_async2()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 760, in _run_async2
    result = await _run_async()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 686, in _run_async
    self._redraw()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 523, in _redraw
    self.context.run(run_in_context)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 509, in run_in_context
    self.renderer.render(self, self.layout)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/renderer.py", line 568, in render
    layout.container.preferred_height(size.columns, size.rows).preferred,
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 792, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 2595, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1620, in preferred_height
    return self._merge_dimensions(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1648, in _merge_dimensions
    preferred = get_preferred()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1613, in preferred_content_height
    return self.content.preferred_height(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 639, in preferred_height
    content = self.create_content(width, height=1)  # Pass a dummy '1' as height.
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 761, in create_content
    document = buffer.document
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 525, in document
    self.text, self.cursor_position, self.selection_state
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 416, in text
    return self._working_lines[self.working_index]
IndexError: list index out of range
Xonsh encountered an issue during launch
Failback to /bin/sh

or

Unhandled exception in event loop:
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 837, in in_term
    await _do_wait_for_enter("Press ENTER to continue...")
  File "/usr/lib/python3.8/contextlib.py", line 178, in __aexit__
    await self.gen.__anext__()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/run_in_terminal.py", line 114, in in_terminal
    app._redraw()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 523, in _redraw
    self.context.run(run_in_context)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 509, in run_in_context
    self.renderer.render(self, self.layout)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/renderer.py", line 568, in render
    layout.container.preferred_height(size.columns, size.rows).preferred,
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 792, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 324, in preferred_height
    dimensions = [
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 325, in <listcomp>
    c.preferred_height(width, max_available_height) for c in self._all_children
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 2595, in preferred_height
    return self.content.preferred_height(width, max_available_height)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1620, in preferred_height
    return self._merge_dimensions(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1648, in _merge_dimensions
    preferred = get_preferred()
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/containers.py", line 1613, in preferred_content_height
    return self.content.preferred_height(
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 639, in preferred_height
    content = self.create_content(width, height=1)  # Pass a dummy '1' as height.
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/layout/controls.py", line 761, in create_content
    document = buffer.document
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 524, in document
    return self._document_cache[
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/cache.py", line 98, in __missing__
    result = self.get_value(*key)
  File "/usr/lib/python3.8/site-packages/prompt_toolkit/document.py", line 98, in __init__
    assert cursor_position is None or cursor_position <= len(text), AssertionError(

Exception cursor_position=48, len_text=3
Press ENTER to continue...

all triggered by the same repro steps (e.g. compose which reveals a long path)!?

[jaraco]

Today the issue happened again, this time I definitely did not hit backspace when typing the characters:

Unhandled exception in event loop:
  File "/Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/asyncio/events.py", line 81, in _run
    self._context.run(self._callback, *self._args)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/application/application.py", line 653, in read_from_input
    self.key_processor.process_keys()
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_processor.py", line 274, in process_keys
    self._process_coroutine.send(key_press)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_processor.py", line 186, in _process
    self._call_handler(matches[-1], key_sequence=buffer[:])
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_processor.py", line 329, in _call_handler
    handler.call(event)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/key_bindings.py", line 101, in call
    self.handler(event)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/key_binding/bindings/named_commands.py", line 270, in self_insert
    event.current_buffer.insert_text(event.data * event.arg)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/buffer.py", line 1209, in insert_text
    self.document = Document(text, cpos)
  File "/Users/jaraco/.local/pipx/venvs/xonsh/lib/python3.8/site-packages/prompt_toolkit/document.py", line 98, in __init__
    assert cursor_position is None or cursor_position <= len(text), AssertionError(

Exception cursor_position=19, len_text=7
Press ENTER to continue...
~ $ cd ~/p/pypa/setuptools

Cursor was after the t in setuptools (position 19 if 1 based, 18 if zero-based).

[hroemer]

@jaraco I think I found the reason for my problems: fairly large history files (~/.local/share/xonsh/*json). The observations stated in #1158 and the related xonsh issue #3586 resemble exactly what I experience. I am curious whether this is the case for you, as well.

[jaraco]

Today, I noticed something. I started up my shell, and before the prompt was presented, I typed a few characters, but then waited for the prompt to appear, which takes a few seconds. After the prompt appeared, I started typing the rest of the command, after which the error appeared. I've noticed the issue is almost always (if not always) at the beginning of a session. These new findings are consistent with the findings mentioned above, so seem to be a likely cause.

My history directory has 4.5k json files totallying about 45MB:

~ $ len(list(pathlib.Path('~/.local/share/xonsh').expanduser().glob('*.json')))
4458

It's too many files to pass to ls ;)

So I think it's a likely trigger.

[jaraco]

Sounds like #1170 is a likely fix.

[jaraco]

I've installed that PR in my main environment and will report back if that doesn't work around the issue.

~ $ ~/.local/pipx/venvs/xonsh/bin/python -m pip install git+https://github.com/bobhy/python-prompt-toolkit@th-threadsafe-load-2