gh-113274: fix EUC-JP decoding of FULLWIDTH TILDE

[qsantos]

This PR closes #113274. This is done by changing the UCS-2 codepoint 126 (~, TILDE) to 65374 (～, FULLWIDTH TILDE) in the __jisx0212_decmap reference table. Since no character is added or removed, no other changes are needed.

Bug report

Bug description:

Python decodes the bytes 0x8FA2A7 as ~ (TILDE) in EUC-JP.

assert b'\x8f\xa2\xb7'.decode('euc_jp') == '~'

This reference document is ambiguous in that it shows a simple ~ (TILDE), but most other software (iconv, Vim, Firefox, Rust's encoding_rs) interpret this as ～ (FULLWIDTH TILDE). Note that EUC-JP already includes US-ASCII, and so:

assert '~'.encode('euc-jp') == b'~'

CPython versions tested on:

3.11, CPython main branch

Operating systems tested on:

Linux

• Issue: Python decodes EUC-JP 8FA2A7 as TILDE instead of FULLWIDTH TILDE #113274

[corona10]

cpython/Modules/cjkcodecs/mappings_jp.h

Line 1 in 171ebb2

// AUTO-GENERATED FILE FROM genmap_japanese.py: DO NOT EDIT

Please take a look at the comment on the first line.
I 've not taken a look at the issue deeply yet, but if you want to change the mapping file, you should modify the generator.

[qsantos]

I totally missed that. I will do that.

[qsantos]

• Python generates the data using this as its source of truth: https://www.unicode.org/Public/MAPPINGS/OBSOLETE/EASTASIA/JIS/JIS0212.TXT.
• encoding_rs: also generated, but from https://encoding.spec.whatwg.org/index-jis0212.txt
• iconv: seems to be maintained indenpendently, and we can actually seen when they replaced TILDE with FULLWIDTH TILDE in their changelog: https://git.savannah.gnu.org/gitweb/?p=libiconv.git;a=blob;f=ChangeLog;h=0e2f5f00bc6aff84932cd92bd09a7c14f802c44d;hb=refs/heads/master#l4729

I couldn't figure out how Vim and Firefox proceed (although Vim does fallback to iconv for non-internally-supported encodings). Since I do not expect to get Unicode to update the document of an obsolete standard that they only used to adapt, I see two options:

1. add a hack in genmap_japanese.py around line 90 after the call to loadmap(jisx0212file) to reassign jisx0212decmap[34][55] = ord('～') (it does fix __jisx0212_decmap as expected and changes nothing else), and comment that properly
2. switch to WhatWG as the source of truth

1 can be done pretty quickly and with minimal risk of breaking anything. 2 has the advantage that it might help discover other issues in mappings, but is obviously much more involved.

If 1 seems reasonable to you, I will update this PR accordingly. If 2, I will close it and document this in the ticket for later.

[qsantos]

I have updated this MR with the hack in genmap_japanese.py and regenerated mappings_jp.h.

[corona10]

I will take a look at which will be the better, If we decide to solve this issue, option 2 will be the better way.
But I need to check the current status related to JIS0212 and side effect.
Also need to consider stability of whatwg spec.

[bedevere-app]

A Python core developer has requested some changes be made to your pull request before we can consider merging it. If you could please address their requests along with any other requests in other reviews from core developers that would be appreciated.

Once you have made the requested changes, please leave a comment on this pull request containing the phrase I have made the requested changes; please review again. I will then notify any core developers who have left a review that you're ready for them to take another look at this pull request.

[qsantos]

I have made the requested changes; please review again.

[corona10]

@qsantos I will take a look the PR by next week. Enjoy your Christmas.

[corona10]

Okay, I like this PR, it will guarantee the round trip

a = b'\x8f\xa2\xb7'
assert a.decode('euc_jp').encode('euc_jp') == a

But as I said, I will take a look at this PR by next week.

[qsantos]

@qsantos I will take a look the PR by next week. Enjoy your Christmas.

Thanks, I was not sure if I had properly triggered the bot. Enjoy your Christmas as well!

[corona10]

Okay, here is my temporal conclusion for this change.
(If we decide to adopt the change!)

1. Let's use uncode.org data as same as now, whatwg is great but not stable. We are not a browser, I am not sure that we can follow every change without side effect.
2. Let's vendor the modified JIS0212.TXT into https://github.com/python/cpython/tree/main/Tools/unicode/python-mappings and store the diff file to https://github.com/python/cpython/tree/main/Tools/unicode/python-mappings/diff for tracking changes.

But I need to listen to the opinion of @ezio-melotti, who is a more experienced unicode expert than me.

[qsantos]

@ezio-melotti What is your take on this?

[qsantos]

Should we consider this dead?

[github-actions]

This PR is stale because it has been open for 30 days with no activity.

[qsantos]

Not stale, just waiting for review

[serhiy-storchaka]

I am fine with patching the generator if it allows us to move forward.

I have only questions: is EUC-JP the only encoding that needs this fix?