Python Launcher for Windows (`py.exe`) breaks on non-python shebang line

[CAD97]

Bug report

Given a non-python shebang line, e.g. #!/usr/bin/env bash, py.exe falls over with

❯ py ./x.py
Unable to create process using '/usr/bin/env bash ./x.py'

py.exe should not be trying to interpret non-python shebang lines on Windows. If py has been invoked (especially if manually invoked rather than implicitly by ftype association), the intent is to interpret the file as a python script.

I originally hit this with rust-lang/rust's x.py script, which is currently using /usr/bin/env bash to attempt to launch Python 3 across all OSes, whether they provide a python executable or just python3. (MSYS makes this more fun, as it does directly interpret and use the shebang line, rather than calling the ftype registered handler.)

Your environment

❯ winget list python
Name            Id                                     Version     Available   Source
-------------------------------------------------------------------------------------
Python 3        Python.Python.3                        3.10.4150.0 3.10.5150.0 winget
Python Launcher {691AAAA1-FE86-4973-8DA2-6AA2B3327562} 3.10.7751.0

Microsoft Windows
Version 21H2 (OS Build 22000.739)

[eryksun]

The launcher implements the virtual shebang "#!/usr/bin/env <name>" to search the directories in the PATH environment variable for the given name. The search is implemented by WinAPI SearchPathW(). If the name has no extension, SearchPathW() is called for each of the PATHEXT extensions in succession until the file is found. For example, the launcher will search for "bash.COM", "bash.EXE", and so on.

Chances are that a developer has "%SystemRoot%\System32\bash.exe" from WSL. However, the launcher is distributed as a 32-bit application, in which case accessing "%SystemRoot%\System32" gets redirected to the 32-bit system directory "%SystemRoot%\SysWOW64", in which "bash.exe" likely doesn't exist.

If the system happens to have bash from an MSYS2 or Cygwin installation in PATH, then that version of the shell will be found.

[CAD97]

And if no bash is available (as is the case on my machine¹), we end up with this error that at the very least reads as if it's trying to run some application named /usr/bin/env. If the shebang is resolved, calling the resolved program is at least reasonable. If the shebang fails to resolve, though, it seems reasonable to (print a warning and) interpret the file with the default python, as running py is a good hint that the user believes the file to be a python script.

Footnotes

1. Time and time again I'm the Windows dev who pops in to say that unixisms can't be assumed to work on Windows. ↩

[eryksun]

There is no difference between a py.exe process that's executed manually versus a file association, so any behavior change for the env virtual command would have to apply to all cases.

For the new implementation of the launcher (3.11+), I think it would be okay to use the default Python when env is passed a non-Python command, if that's not already the case. (I haven't had time to review most of the new launcher code in "PC/launcher2.c".) The new behavior could also be corrected to search for "python" commands that include a version specifier, i.e. "python<version><.exe>" (e.g. "python3.11.exe"). SearchPathW() doesn't reliably append the optional lpExtension value (e.g. it will append it to ".\python3.11" but not to "python3.11"), so the ".exe" extension has to be added manually if it's missing. Fall back on a registered command if SearchPathW() fails.

For the old launcher, it at least shouldn't try to pass the env command to CreateProcessW(). The launcher should fail if the search fails, and the error message should be clear. The user may not know whether or how the launcher supports the env virtual command. The documentation doesn't specify the behavior except for env python.

[CAD97]

For the new implementation of the launcher (3.11+), I think it would be okay to use the default Python when env is passed a non-Python command, if that's not already the case. (I haven't had time to review most of the new launcher code in "PC/launcher2.c".)

I glanced at the implementation; launcher2.c falls back to the no-shebang behavior if the shebang is not a python shebang or a registered py.ini command. So the new launcher already fixes this issue / doesn't implement this unexpected behavior 🎉

cpython/PC/launcher2.c

Lines 900 to 921 in 68fb032

	} else if (_findCommand(search, command, commandLength)) {
	search->executableArgs = &command[commandLength];
	search->executableArgsLength = shebangLength - commandLength;
	debug(L"# Treating shebang command '%.*s' as %s\n",
	commandLength, command, search->executablePath);
	} else if (_shebangStartsWith(command, commandLength, L"python", NULL)) {
	search->tag = &command[6];
	search->tagLength = commandLength - 6;
	search->oldStyleTag = true;
	search->executableArgs = &command[commandLength];
	search->executableArgsLength = shebangLength - commandLength;
	if (search->tag && search->tagLength) {
	debug(L"# Treating shebang command '%.*s' as 'py -%.*s'\n",
	commandLength, command, search->tagLength, search->tag);
	} else {
	debug(L"# Treating shebang command '%.*s' as 'py'\n",
	commandLength, command);
	}
	} else {
	debug(L"# Found shebang command but could not execute it: %.*s\n",
	commandLength, command);
	}

The user may not know whether or how the launcher supports the env virtual command. The documentation doesn't specify the behavior except for env python.

Which is actually part of my argument that py should assume the script is a python script, and not try to implement env. I know that I at least as a Windows user, where ftype is determined solely by extension/metadata and not by shebang, would expect running ./x.py or especially py ./x.py to always use the python interpreter.

Even on a unixy platform where a shebang is respected for ./x.py, a user can still run e.g. python3 ./x.py, which will interpret as python and not try to dispatch to the shebang-specified interpreter. Any *.py file which cannot be interpreted as Python is going to cause problems one way or another.

The new behavior could also be corrected to search for "python" commands that include a version specifier

launcher2.c already does implement this behavior. A shebang of (vcommand) python{version} will be sliced directly up into calling py -{version}.

SearchPathW() doesn't reliably append the optional lpExtension value

It's documented as "the extension is added only if the specified file name does not end with an extension," so yeah, if the base name contains a ., the caller has to recognize that it's not an extension and append the proper extension manually. (And I'm surprised that .\python3.11 functions for anything at all, given that is not a file name, and in this API you pass the path separately.)

For the old launcher, it at least shouldn't try to pass the env command to CreateProcessW(). The launcher should fail if the search fails, and the error message should be clear.

I just checked the code, and I think this line is the offending one that puts the full shebang line into the command buffer to be run. Just removing that should make maybe_handle_shebang fall through and return (signalling that no shebang is present). Alternatively (or additionally), an additional outparam to parse_shebang is needed for when cp == NULL signals that command lookup failed.

[eryksun]

Which is actually part of my argument that py should assume the script is a python script, and not try to implement env.

It's a bad idea to change the basic behavior of the old launcher, which has been in place for a long time. But it seems reasonable to clean it up to not call CreateProcessW() if the env search fails and to provide a clearer error message in that case.

Even on a unixy platform where a shebang is respected for ./x.py, a user can still run e.g. python3 ./x.py, which will interpret as python and not try to dispatch to the shebang-specified interpreter.

The launcher wasn't implemented to take a private parameter when it's invoked for a file association, as opposed to explicitly invoked on the command line, so there's no way to reliably distinguish these cases for shebang handling. That said, if you specify a version, the shebang is ignored, e.g. run py -3 .\x.py.

It's documented as "the extension is added only if the specified file name does not end with an extension," so yeah, if the base name contains a ., the caller has to recognize that it's not an extension and append the proper extension manually. (And I'm surprised that .\python3.11 functions for anything at all, given that is not a file name, and in this API you pass the path separately.)

SearchPathW() can be called with qualified paths. This occurs all the time, e.g. as called by the internal implementation of CreateProcessW(). In a search context in Windows, partially qualified paths usually aren't resolved solely (if at all) against the current working directory. For example, "spam\python" will be resolved against each directory in the lpPath search path until a match is found. SearchPathW() will try to append lpExtension only if the name has no extension. The exception, however, is a filename that explicitly refers to the current working directory by beginning with a "." or ".." component, such as ".\python3.11" or "..\python3.11". These cases are resolved only against the current working directory. This is the inconsistent case, because SearchPathW() will try appending lpExtension regardless of whether lpFileName already has an extension (e.g. when searching for ".\python3.11", it can find ".\python3.11.exe" if ".exe" is the optional extension). We have to avoid this inconsistency, and we need to support base filenames that Windows sees as having an extension, e.g. "python3.11". Thus the ".exe" extension must be added manually instead of relying on lpExtension.

[CAD97]

The user may not know whether or how the launcher supports the env virtual command. The documentation doesn't specify the behavior except for env python.

Not to hurt my own case, but

The /usr/bin/env form of shebang line has one further special property. Before looking for installed Python interpreters, this form will search the executable PATH for a Python executable. This corresponds to the behaviour of the Unix env program, which performs a PATH search.

This could be read as documenting that #! /usr/bin/env x searches PATH for the (assumed to be) python executable x. I'd argue that it doesn't, and only the four listed forms are documented as supported, but in the face of the existing implementation, that documentation can be argued to refer to that behavior.

[eryksun]

This could be read as documenting that #! /usr/bin/env x searches PATH for the (assumed to be) python executable x. I'd argue that it doesn't, and only the four listed forms are documented as supported, but in the face of the existing implementation, that documentation can be argued to refer to that behavior.

I don't think it was the intention to document support for anything but a "python" executable, with some supported extension from PATHEXT (".com", ".exe", etc), per the listed case "/usr/bin/env python". However, the old launcher doesn't limit itself to the documented behavior. It also has strange behavior in refusing to search PATH for versioned "python" names such as "python3", while happily searching for "bash".

[CAD97]

While I should have gathered this from when I skimmed the code: the path lookup behavior isn't limited to /usr/bin/env, but happens if you just write a non-python name. This means that #! py will happily result in an infinite loop of recursively creating new py processes 😄

[zooba]

If I'm reading this right, I accidentally fixed everything in the new launcher and we don't need this issue anymore? (That's my favourite kind of fix.)

This means that #! py will happily result in an infinite loop of recursively creating new py processes

Yep 😄 Might be worth catching that specific case, but py -3.10 should be fine, so it's not entirely trivial. Spelling it *nix style is better, of course, if only because it'll work on more platforms.

[CAD97]

If I'm reading this right, I accidentally fixed everything in the new launcher and we don't need this issue anymore? (That's my favourite kind of fix.)

Yes. As the new py.exe is currently implemented, it doesn't implement the undocumented behavior of the old py.exe of interpreting shebangs that aren't for python{version}, so means that it will just call #!/usr/bin/env bash with the python interpreter.

(This is imho the correct behavior. Calling py means "launch python, matching the file-requested version", not "launch via shebang, unless there isn't a shebang, in which case use python.")

[eryksun]

If I'm reading this right, I accidentally fixed everything in the new launcher and we don't need this issue anymore?

The new launcher doesn't implement the search semantics of "/usr/bin/env python". The behavior is documented as follows:

The /usr/bin/env form of shebang line has one further special property. Before looking for installed Python interpreters, this form will search the executable PATH for a Python executable. This corresponds to the behaviour of the Unix env program, which performs a PATH search.

In the old launcher, the virtual "env" command is implemented to work like a PATH search in a CLI shell, by calling SearchPathW() for all PATHEXT extensions. (CreateProcessW() itself calls SearchPathW(), but only with ".EXE" as the optional extension, not all PATHEXT extensions.) The old launcher doesn't search PATH for versioned "python" commands such as "python3" or "python3.10", though I always thought it should, with care taken to add the ".exe" extension for names such as "python3.10" (i.e. ".10" is not an extension). The old launcher's "env" virtual command also searches for any command that doesn't start with "python", which is undocumented behavior that the new launcher doesn't have to implement. I say it's undocumented because the documentation refers to the " /usr/bin/env form of shebang line", which is only listed as "/usr/bin/env python".

[zooba]

Yeah, from a security POV I'm happier not letting the launcher launch any application just because someone double-clicked on a .py file (though of course that file may well launch it, and likely more reliably, but easier to detect).

At the same time I don't want to exclude non-CPython runtimes from being detected, and only allowing (effectively) python.exe (or PATHEXT) to be searched could do that.

I think on balance, I'm coming down on the "run anything" path, and we recommend against installing or using the launcher in secure contexts. Which I normally recommend against anyway, just to remove a few potential hijacks, so the advice won't change.

To be clear about the semantics:

• when the shebang line matches /usr/bin/env (\S+)(.+)
• search each directory on PATH for the first capture, following SearchPath semantics
• if found, pretend it's a python.exe-like CLI and run it
• if not, continue as we do today (which means failing if the first capture isn't python.*