RFE: Unicode 18.0 support

[apnadkarni]

Title says it all - a request for Unicode 18.0 support. The new version of the standard is due out in September. Any plans to support it?

Currently, graphemetest fails on some 16 odd tests with the 18.0 version of GraphemeBreakTest.txt. (This is after rebuilding utf8proc_data.c by changing the Unicode version in data/Makefile to 18)

For example,

D:\src\utf8proc\build\Debug>graphemetest.exe GraphemeBreakTest.txt
checking line 100...
FAILED at line 179: grapheme mismatch: "/्/क" instead of "/्क"

And BTW, thanks much for this library which we use in the Tcl 9 language core.

[eschnett]

We are all eagerly awaiting the release of the new version of the standard. We have the usual (open-source we-quite-want-it but we're-not-paid-to-provide-this) commitment to support it.

[stevengj]

Usually graphemetest failures means that they changed the grapheme rules (yet again), so we'll need to update some of the logic in addition to downloading the new data tables.

[apnadkarni]

Thanks for the quick response. I did manage to get failures down to three test cases (from a score or so) but honestly, no idea what I'm doing and got lost at this rule

[stevengj]

According to the draft Unicode 18.0.0 release notes, this was the change to the grapheme-break rules (affecting Indic languages only):

The rule looks a lot simpler now, so our code for indic conjunct breaks (added in #253) should probably simplify.

and the definition of InCB=Linker was also changed, apparently:

but that change should already be in the new DerivedCoreProperties.txt.

[stevengj]

As I understand it, the change to utf8proc.c to implement the revised rule should simply be:

diff --git a/utf8proc.c b/utf8proc.c
index ad3cae9..7fee840 100644
--- a/utf8proc.c
+++ b/utf8proc.c
@@ -101,7 +101,7 @@ UTF8PROC_DLLEXPORT const char *utf8proc_version(void) {
 }
 
 UTF8PROC_DLLEXPORT const char *utf8proc_unicode_version(void) {
-  return "17.0.0";
+  return "18.0.0";
 }
 
 UTF8PROC_DLLEXPORT const char *utf8proc_errmsg(utf8proc_ssize_t errcode) {
@@ -294,7 +294,7 @@ static utf8proc_bool grapheme_break_extended(int lbc, int tbc, int licb, int tic
     int state_bc, state_icb; /* boundclass and indic_conjunct_break state */
     if (*state == 0) { /* state initialization */
       state_bc = lbc;
-      state_icb = licb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT ? licb : UTF8PROC_INDIC_CONJUNCT_BREAK_NONE;
+      state_icb = licb;
     }
     else { /* lbc and licb are already encoded in *state */
       state_bc = *state & 0xff;  // 1st byte of state is bound class
@@ -305,16 +305,9 @@ static utf8proc_bool grapheme_break_extended(int lbc, int tbc, int licb, int tic
        !(state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER
         && ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT); // GB9c
 
-    // Special support for GB9c.  Don't break between two consonants
-    // separated 1+ linker characters and 0+ extend characters in any order.
-    // After a consonant, we enter LINKER state after at least one linker.
-    if (ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT
-        || state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_CONSONANT
-        || state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND)
-      state_icb = ticb;
-    else if (state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER)
-      state_icb = ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND ?
-                  UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER : ticb;
+    // Special support for GB9c.  Don't break between linker + 0+ extend chars and consonant.
+    // We enter LINKER state after a linker and stay in it for extend chars.
+    state_icb = (state_icb == UTF8PROC_INDIC_CONJUNCT_BREAK_LINKER && ticb == UTF8PROC_INDIC_CONJUNCT_BREAK_EXTEND) ? state_icb : ticb;
 
     // Special support for GB 12/13 made possible by GB999. After two RI
     // class codepoints we want to force a break. Do this by resetting the

However, I'm getting a test failure:

FAILED at line 883: grapheme mismatch: "/्̀̀त/क" instead of "/्̀̀/त/क"

But if I read the new rule correctly, it seems that "/्̀̀त/क" is correct?

[apnadkarni]

Great progress!

I don't see that line 883 in my GraphemeBreakTest.txt for 18.0 (dated 5/12) which is the latest that I could find.

[stevengj]

Oh, that failure actually isn't from GraphemeBreakTest.txt, it's from our manual GB9c tests here, which clearly need to be updated:

utf8proc/test/graphemetest.c

Lines 123 to 128 in 74e0b22

	/* more GB9c tests */
	checkline("/ 0915 0300 094d 0300 0924 / 0915 /", true);
	checkline("/ 0915 0300 094d 0300 094d 0924 / 0915 /", true);
	checkline("/ 0915 0300 0300 / 0924 / 0915 /", true);
	checkline("/ 0915 0300 094d 0300 / 0078 /", true);
	checkline("/ 0300 094d 0300 / 0924 / 0915 /", true);

I fixed the test, and now make check is passing. I've created a draft PR #345.