Python cannot start in locale zh_TW.euctw

[serhiy-storchaka]

$ LC_ALL=zh_TW.euctw ./python
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './python'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  is in build tree = 1
  stdlib dir = '/home/serhiy/py/cpython/Lib'
  sys._base_executable = '/home/serhiy/py/cpython/python'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/home/serhiy/py/cpython/python'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python311.zip',
    '/home/serhiy/py/cpython/Lib',
    '/home/serhiy/py/cpython/build/lib.linux-x86_64-3.11-pydebug',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
LookupError: unknown encoding: EUC-TW

Current thread 0x00007f1f24b4b280 (most recent call first):
  <no Python frame>

It is one of standard glibc locales. Other unsupported locales are hy_AM.armscii8 and ka_GE.georgianps, but I suppose they are less used, and there is no much official documentation available for that encodings.

[edit @StanFromIreland/@encukou] Another unsupported locale is IBM-943 (Japanese shift_jis) (#130228)

[methane]

Maybe, enable UTF-8 Mode when locale encoding is not found?

[vstinner]

https://peps.python.org/pep-0686/ may fix the issue :-) cc @methane

[StanFromIreland]

Another instance of this #130228 with the IBM-943 (Japanese shift_jis) locale

[StanFromIreland]

I can get it to work forcing it to utf-8 in:

cpython/Python/initconfig.c

Lines 2276 to 2280 in 7473c60

	// See also config_get_fs_encoding()
	static PyStatus
	config_get_locale_encoding(PyConfig *config, const PyPreConfig *preconfig,
	wchar_t **locale_encoding)
	{

I could implement a fallback, but the question is, is there even a point? PEP 686 will force it to UTF8 if I am reading it correctly.

So, do we want fallback, or could we just implement the PEP now?

cc @methane

And do we also want a SC decision?

[methane]

@StanFromIreland PEP 686 makes UTF-8 Mode default since Python 3.15.
Until then, user can enable UTF-8 Mode by export PYTHONUTF8=1 in .bashrc.

[StanFromIreland]

Duplicate: #63658

[StanFromIreland]

PEP 686 makes UTF-8 Mode default since Python 3.15.
Until then, user can enable UTF-8 Mode by export PYTHONUTF8=1 in .bashrc.

I know, that is why I cc'ed you. This issue will be closed when that is implemented, but we are on 3.14 and not 3.15, do we still want a workaround here or do we want to just wait or implement it already?

[methane]

It depends on how 'zh_TW.euctw' is used. If there are some users in the wild, we should add it to alias.
For example, #131370 (comment)