The return value of mb_list_encodings() doesn't contain 'SJIS-win'

[sji-il]

Description

The following code:

<?php
var_dump(mb_list_encodings());

https://3v4l.org/GH0Gj

Resulted in this output:

• contains 'CP932' and 'SJIS', but no 'SJIS-win'

But I expected this output instead:

• contains 'SJIS-win'

As of PHP 8.1, SJIS-win is an alias of CP932, thus the return value of mb_list_encodings() doesn't contain 'SJIS-win'.

This breaks BC.

Some libraries have functions which receive one or more encoding names as arguments, and they often validate the input by the return value of mb_list_encodings().
This should be reverted, or fixed, or at least documented as an incompatible change.

PHP Version

PHP 8.1.0

Operating System

No response

[alexdowad]

@sji-il, thank you for the report.

This change actually dates back to 2020; see e245985. Before that commit, SJIS-win was indeed treated as a separate encoding from CP932, but testing revealed that the behavior of SJIS-win and CP932 was exactly the same (i.e. all the same mappings).

I think the first question here is: Was there actually some difference between SJIS-win and CP932 which I missed? Perhaps more importantly, should there have been some difference?

If indeed there is no difference, do you still feel that mb_list_encodings should list the same legacy text encoding twice, under two different names? Or perhaps SJIS-win should be the primary name, and CP932 should be an alias?

Regarding the BC issue, I'm sorry that this was not documented as a backwards-incompatible change. I think this is actually a change from PHP 7.4 to 8.0, not from 8.0 to 8.1 (but please correct me if that is wrong).

[cmb69]

SJIS-win is only an alias of CP932 as of PHP 8.1.0 (https://3v4l.org/bF8G4). I think we should document this change, but not revert it.

Some libraries have functions which receive one or more encoding names as arguments, and they often validate the input by the return value of mb_list_encodings().

These libraries probably should also check mb_encoding_aliases() (something like https://www.php.net/manual/en/function.mb-list-encodings.php#122266).

[sj-i]

@alexdowad

Was there actually some difference between SJIS-win and CP932

https://3v4l.org/AVllX
This code behaves differently between SJIS-win and CP932 before 8.1.

should there have been some difference?

As there has been some difference already 20+ years... there should be, unfortunately.
Yeah, I think I can hear the third sigh from you.
Legacy Japanese encodings should all be burned by fire and buried in the dark, though it's not realistic.
SJIS-win and CP932 are both used in Japan. Some articles recommend one, others recommend another. It is optimistic to say that keeping only one behavior is enough. Both could be depended on by a fair number of users.

mb_list_encodings should list the same legacy text encoding twice

Even if there is no difference, I would recommend keeping two names on the list to avoid BC breaks, unless the maintenance burden is too high. When the maintenance burden is so much problematic, changing it in a major release with deprecation beforehand is reasonable. If the behavior was newborn or clearly disadvantageous to users, I feel it could simply be fixed as a bug in a minor release though...

this is actually a change from PHP 7.4 to 8.0, not from 8.0 to 8.1

It's from PHP 8.1.
https://3v4l.org/glAfa

[alexdowad]

@sj-i, thank you for pointing out the difference in mappings between SJIS-win and CP932. I don't know where I messed up when testing mbstring's handling of these text encodings back in 2020, but in any case, the correction is appreciated. (As mentioned, when testing, it appeared that there was no difference in behavior.)

I would just like to ask: as far as you know, is U+203E the only codepoint which should be handled differently between SJIS-win and CP932? Or are there any others?

[alexdowad]

(Commenting further on the sample code which @sj-i shared:) Hmm. This actually seems to be closely related to the same issue which you have commented on in #8281. (Whether SJIS 0x7E should be interpreted as U+007E or U+203E...)

It does appear that the legacy behavior of CP932 in mbstring was a bit illogical, in that when converting Unicode → CP932, it would convert U+007E to 0x7E (implying that 0x7E is being used to represent a tilde, not an overbar), but then when converting CP932 → Unicode, it would convert U+203E to 0x7E (implying that 0x7E represents an overbar, not a tilde).

Do you have any idea why this is the case?

SJIS-win and CP932 are both used in Japan. Some articles recommend one, others recommend another. It is optimistic to say that keeping only one behavior is enough. Both could be depended on by a fair number of users.

If these encodings are different, and both are used by even a small number of users, then we definitely need to support both of them. That is not under question. Currently I am still trying to understand clearly what the difference is.

[sj-i]

@alexdowad

Or are there any others?

I can't say sure, but this is the only one I'm aware of.

Do you have any idea why this is the case?

I'm not sure about this specific case.
Tilde (0x7E in ASCII) is often used in programming languages for special purposes. So historically there is confusion about how 0x7E in legacy encodings should be converted to Unicode, and how its roundtrip should be treated.

[sj-i]

@alexdowad
https://3v4l.org/lQpnG
I found that conversion from U+00A5 is also different between CP932 and SJIS-win via brute-forcing.

I've also asked the original author, and he said that SJIS-win is a compatibility implementation created for historical reasons to map U+00A5 and U+203E in the same way as eucJP-win, so that the differences with CP932 are intended and should be kept.

[alexdowad]

I've also asked the original author, and he said that SJIS-win is a compatibility implementation created for historical reasons to map U+00A5 and U+203E in the same way as eucJP-win, so that the differences with CP932 are intended and should be kept.

Understood. 👍🏻

So it looks like the solution to this issue here is to restore SJIS-win. That will also make it 'reappear' in mb_list_encodings.

Will do that as soon as I have a bit of time to prepare a PR.

[alexdowad]

Just did a first draft of the code to reintroduce SJIS-win (it's just a few lines). Still need to adjust the unit tests.

I also still need to re-examine the mappings for CP932, confirm if there have been any changes from legacy mbstring, and if so, raise each one for discussion here.

I think it would be helpful if we can include some explanation of why users might choose to use CP932 or SJIS-win in the code comments; this would help future maintainers and developers to understand why the code is the way it is.

@sj-i stated above:

SJIS-win and CP932 are both used in Japan. Some articles recommend one, others recommend another.

I've not yet read through the references provided by @zonuexe on #8281, so maybe this is already covered there, but can you explain any reasons why one might recommend the use of CP932, or conversely, might recommend SJIS-win?

[alexdowad]

Patch is ready for review, just waiting to see if @sj-i or any other interested parties can provide any additional background information which can go into code comments.