1/* Part of SWI-Prolog 2 3 Author: Jan Wielemaker 4 E-mail: J.Wielemaker@vu.nl 5 WWW: http://www.swi-prolog.org 6 Copyright (c) 2010-2026, University of Amsterdam 7 CWI, Amsterdam 8 SWI-Prolog Solutions b.v. 9 All rights reserved. 10 11 Redistribution and use in source and binary forms, with or without 12 modification, are permitted provided that the following conditions 13 are met: 14 15 1. Redistributions of source code must retain the above copyright 16 notice, this list of conditions and the following disclaimer. 17 18 2. Redistributions in binary form must reproduce the above copyright 19 notice, this list of conditions and the following disclaimer in 20 the documentation and/or other materials provided with the 21 distribution. 22 23 THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS 24 "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT 25 LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS 26 FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE 27 COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, 28 INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, 29 BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; 30 LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 31 CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT 32 LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN 33 ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE 34 POSSIBILITY OF SUCH DAMAGE. 35*/ 36 37:- module(unicode, 38 [ unicode_property/2, % ?Code, ?Property 39 unicode_map/3, % +In, -Out, +Options 40 unicode_nfd/2, % +In, -Out 41 unicode_nfc/2, % +In, -Out 42 unicode_nfkd/2, % +In, -Out 43 unicode_nfkc/2, % +In, -Out 44 unicode_nfkc_casefold/2, % +In, -Out 45 unicode_casefold/2, % +In, -Out 46 unicode_version/1, % -Version 47 unicode_codepoint_valid/1, % +Code 48 atom_graphemes/2, % ?Atom, ?Graphemes 49 string_graphemes/2 % ?String, ?Graphemes 50 ]). 51:- use_foreign_library(foreign(unicode4pl)). 52 53/** <module> Unicode string handling 54 55This library wraps the [utf8proc](https://github.com/JuliaStrings/utf8proc) 56library, giving Prolog code access to Unicode character properties, 57string normalization, case folding, and grapheme-cluster iteration. 58 59Three levels of API are provided: 60 611. **Normalization**: unicode_nfd/2, unicode_nfc/2, unicode_nfkd/2, 62 unicode_nfkc/2 implement the four standard Unicode normalization 63 forms (NFD, NFC, NFKD, NFKC; see UAX#15). unicode_nfkc_casefold/2 64 combines NFKC with case folding for caseless identifier matching 65 (see UAX#31). 662. **Per-codepoint properties**: unicode_property/2 queries the 67 Unicode property database (general category, bidi class, 68 decomposition type, display width, case mappings, grapheme 69 boundary class, ...). 703. **Mixed string-level transformations**: unicode_map/3 is the 71 workhorse; it accepts a list of flags chosen from a fixed set and 72 performs the corresponding composition of decompose / compose / 73 strip / lump / case-fold / grapheme-boundary-mark operations in a 74 single pass. unicode_casefold/2 is a convenience wrapper. 75 76Grapheme clusters (user-perceived characters) can be iterated with 77atom_graphemes/2 and string_graphemes/2. 78 79Loading this library also installs a Unicode NFC normalisation hook 80into the SWI-Prolog kernel. The kernel's `unicode_atoms` policy 81(Prolog flag, stream property and `read_term/2,3` option) uses this 82hook for its `nfc` and `error` modes; without the library loaded 83those modes raise `existence_error(hook, unicode_normalize)`. The 84kernel's quoted-write rule that force-quotes atoms containing 85combining marks is independent and works even without this library 86loaded. 87 88Lump handling: 89 90== 91U+0020 <-- all space characters (general category Zs) 92U+0027 ' <-- left/right single quotation mark U+2018..2019, 93 modifier letter apostrophe U+02BC, 94 modifier letter vertical line U+02C8 95U+002D - <-- all dash characters (general category Pd), 96 minus U+2212 97U+002F / <-- fraction slash U+2044, 98 division slash U+2215 99U+003A : <-- ratio U+2236 100U+003C < <-- single left-pointing angle quotation mark U+2039, 101 left-pointing angle bracket U+2329, 102 left angle bracket U+3008 103U+003E > <-- single right-pointing angle quotation mark U+203A, 104 right-pointing angle bracket U+232A, 105 right angle bracket U+3009 106U+005C \ <-- set minus U+2216 107U+005E ^ <-- modifier letter up arrowhead U+02C4, 108 modifier letter circumflex accent U+02C6, 109 caret U+2038, 110 up arrowhead U+2303 111U+005F _ <-- all connector characters (general category Pc), 112 modifier letter low macron U+02CD 113U+0060 ` <-- modifier letter grave accent U+02CB 114U+007C | <-- divides U+2223 115U+007E ~ <-- tilde operator U+223C 116== 117 118@see http://www.unicode.org/reports/tr15/ (UAX#15 Normalization) 119@see http://www.unicode.org/reports/tr29/ (UAX#29 Grapheme Clusters) 120@see http://www.unicode.org/reports/tr31/ (UAX#31 Identifiers) 121@see https://github.com/JuliaStrings/utf8proc 122*/ 123 124systemgoal_expansion(unicode_map(In, Out, Options), 125 unicode_map(In, Out, Mask)) :- 126 is_list(Options), 127 unicode_option_mask(Options, Mask). 128 129%! unicode_map(+In, -Out, +Options) is det. 130% 131% Perform a Unicode mapping on In, returning Out. Options is a list 132% that may contain any combination of the flags below; a call is 133% roughly equivalent to `utf8proc_map(In, Options)` in the C API. 134% 135% * stable 136% Respect Unicode versioning stability --- the result does not 137% depend on which (recent) version of Unicode is in use. 138% * compat 139% Use compatibility decomposition (i.e. formatting information is 140% lost). 141% * compose 142% Produce a composed result (e.g. NFC or NFKC, depending on the 143% presence of `compat`). 144% * decompose 145% Produce a decomposed result (NFD/NFKD). 146% * ignore 147% Strip "default ignorable" characters (e.g. soft hyphen, zero-width 148% space). 149% * rejectna 150% Raise an error instead of returning output when the input contains 151% unassigned code points. 152% * nlf2ls 153% Convert all NLF-sequences (LF, CRLF, CR, NEL) to U+2028 LINE 154% SEPARATOR. 155% * nlf2ps 156% Convert all NLF-sequences to U+2029 PARAGRAPH SEPARATOR. 157% * nlf2lf 158% Convert all NLF-sequences to U+000A LINE FEED. 159% * stripcc 160% Strip or convert control characters. NLF-sequences become a 161% space, except if one of the NLF-conversion flags is set; HT and 162% FF are treated as NLF in this case. All other control 163% characters are removed. 164% * casefold 165% Apply Unicode case folding (for caseless comparison). 166% * charbound 167% Insert a U+00FF byte at the beginning of every grapheme cluster 168% (UAX#29). The result can be split on 0xFF to recover individual 169% graphemes; atom_graphemes/2 wraps this pattern. 170% * lump 171% Normalise typographic variants to their ASCII equivalents 172% (see module header for the full list). Combined with 173% `nlf2lf`, paragraph and line separators become U+000A as well. 174% * stripmark 175% Strip all combining marks (non-spacing, spacing, enclosing). 176% Must be combined with `compose` or `decompose`. 177 178%! unicode_nfd(+In, -Out) is det. 179% 180% Characters in In are decomposed by canonical equivalence (NFD). 181% Precomposed characters expand into base + combining marks. For 182% example U+00C5 (LATIN CAPITAL LETTER A WITH RING ABOVE) becomes 183% the two-code sequence `A` + U+030A. 184% 185% @see http://www.unicode.org/reports/tr15/ 186 187unicode_nfd(In, Out) :- 188 unicode_map(In, Out, [stable,decompose]). 189 190%! unicode_nfc(+In, -Out) is det. 191% 192% Characters in In are decomposed and then recomposed by canonical 193% equivalence (NFC). Precomposed code points are preferred; for 194% example `A` + U+030A becomes the single code point U+00C5. 195% 196% @see http://en.wikipedia.org/wiki/Unicode_equivalence#Normal_forms 197 198unicode_nfc(In, Out) :- 199 unicode_map(In, Out, [stable,compose]). 200 201%! unicode_nfkd(+In, -Out) is det. 202% 203% Characters in In are decomposed by compatibility equivalence 204% (NFKD). Compatibility decomposition expands presentation forms 205% (ligatures, subscripts, fullwidth letters) into their base forms; 206% for example U+FB03 (LATIN SMALL LIGATURE FFI) becomes `f f i`. 207 208unicode_nfkd(In, Out) :- 209 unicode_map(In, Out, [stable,decompose,compat]). 210 211%! unicode_nfkc(+In, -Out) is det. 212% 213% Characters in In are decomposed by compatibility equivalence, then 214% recomposed by canonical equivalence (NFKC). 215 216unicode_nfkc(In, Out) :- 217 unicode_map(In, Out, [stable,compose,compat]). 218 219%! unicode_nfkc_casefold(+In, -Out) is det. 220% 221% Equivalent to unicode_nfkc/2 followed by unicode_casefold/2 done 222% in a single pass. This is the normalisation form recommended by 223% UAX#31 for caseless identifier matching. For example German 224% `'Strasse'` written with U+00DF (LATIN SMALL LETTER SHARP S) 225% maps to `'strasse'`, and U+FB03 (LATIN SMALL LIGATURE FFI) maps 226% to `'ffi'`. 227 228unicode_nfkc_casefold(In, Out) :- 229 unicode_map(In, Out, [stable,compose,compat,casefold]). 230 231%! unicode_casefold(+In, -Out) is det. 232% 233% Out is the case-folded form of In. Use this for caseless 234% comparison that does not require NFKC; otherwise prefer 235% unicode_nfkc_casefold/2. 236 237unicode_casefold(In, Out) :- 238 unicode_map(In, Out, [stable,casefold]). 239 240 241%! unicode_property(?Code, ?Property) is nondet. 242% 243% Query the Unicode character database for Code. Code is an integer 244% code point (0 .. 0x10FFFF) or a single-character atom; Property is 245% a term of the form Name(Value) drawn from the list below. 246% 247% This predicate is a thin wrapper over utf8proc's property struct, 248% so its vocabulary matches the utf8proc documentation. In the 249% modes (+,?) and (-,?) the predicate enumerates properties for 250% the given code (or the code for the given property); in (+,+) it 251% is a deterministic test. 252% 253% Supported properties: 254% 255% * category(Atom) 256% Unicode general category. Atom is one of `Cc`, `Cf`, `Cn`, 257% `Co`, `Cs`, `Ll`, `Lm`, `Lo`, `Lt`, `Lu`, `Mc`, `Me`, `Mn`, 258% `Nd`, `Nl`, `No`, `Pc`, `Pd`, `Pe`, `Pf`, `Pi`, `Po`, `Ps`, 259% `Sc`, `Sk`, `Sm`, `So`, `Zl`, `Zp`, `Zs`. When querying, the 260% single capital letter of a subcategory stands for all its 261% subcategories; e.g. 262% 263% == 264% ?- unicode_property(0'A, category('L')). 265% true. 266% == 267% 268% * combining_class(Integer) 269% Canonical combining class (0 for base characters, 230 for 270% accents above, etc.). 271% * bidi_class(Atom) 272% Bidirectional class. One of `l`, `lre`, `lro`, `r`, `al`, 273% `rle`, `rlo`, `pdf`, `en`, `es`, `et`, `an`, `cs`, `nsm`, 274% `bn`, `b`, `s`, `ws`, `on`. 275% * bidi_mirrored(Bool) 276% `true` if the character is mirrored for bidi (parentheses, 277% brackets, math operators, ...). 278% * decomp_type(Atom) 279% Compatibility decomposition type. One of `font`, `nobreak`, 280% `initial`, `medial`, `final`, `isolated`, `circle`, `super`, 281% `sub`, `vertical`, `wide`, `narrow`, `small`, `square`, 282% `fraction`, `compat`. Fails when there is no decomposition. 283% * ignorable(Bool) 284% `true` if the character is a "default ignorable" code point. 285% * boundclass(Atom) 286% UAX#29 grapheme-cluster break class. One of `start`, 287% `other`, `cr`, `lf`, `control`, `extend`, `l`, `v`, `t`, 288% `lv`, `lvt`, `regional_indicator`, `spacingmark`, `prepend`, 289% `zwj`, `extended_pictographic`, `e_zwg`. 290% * width(Integer) 291% Display width in fixed-width cells, 0..3. Zero for combining 292% marks and control characters, 1 for most, 2 for "wide" 293% characters (CJK, emoji). 294% * ambiguous_width(Bool) 295% `true` if the character has East-Asian Ambiguous width --- 296% normally one column, but two in a legacy CJK context. 297% * uppercase(Code) 298% * lowercase(Code) 299% * titlecase(Code) 300% Single-code-point case mapping. Fails when the code point 301% has no mapping of that kind (e.g. `unicode_property(0'A, 302% uppercase(_))` fails because `'A'` is already upper-case). 303% For characters whose case mapping produces more than one code 304% point (e.g. U+00DF LATIN SMALL LETTER SHARP S maps to "SS"), 305% use unicode_map/3 with the `[casefold]` option or 306% unicode_casefold/2 for a full string-level transformation. 307% * indic_conjunct_break(Atom) 308% Indic_Conjunct_Break property (Unicode 15+; UAX#44). One of 309% `none`, `linker`, `consonant`, `extend`. Used by the 310% grapheme-cluster-break algorithm for Devanagari, Bengali, 311% etc. 312% 313% @see http://www.unicode.org/reports/tr44/ (Unicode property database) 314 315unicode_property(Code, Property) :- 316 nonvar(Code), nonvar(Property), 317 !, 318 '$unicode_property'(Code, Property, false). 319unicode_property(Code, Property) :- 320 nonvar(Code), 321 !, 322 property(Property), 323 '$unicode_property'(Code, Property, true). 324unicode_property(Code, Property) :- 325 var(Code), 326 !, 327 between(0, 0x10ffff, Code), 328 property(Property), 329 '$unicode_property'(Code, Property, true). 330 331% The third argument of `$unicode_property/3` is a `Silent` flag. 332% When `true`, the C layer returns `false` instead of raising 333% `domain_error(unicode_property, ...)` for a property name the 334% binding was compiled without (older libutf8proc versions lack 335% some fields). Enumeration modes set it to `true` so 336% property/1 clauses for absent features are silently skipped. 337 338property(category(_)). 339property(combining_class(_)). 340property(bidi_class(_)). 341property(bidi_mirrored(_)). 342property(decomp_type(_)). 343property(ignorable(_)). 344property(boundclass(_)). 345property(width(_)). 346property(ambiguous_width(_)). 347property(uppercase(_)). 348property(lowercase(_)). 349property(titlecase(_)). 350property(indic_conjunct_break(_)). 351 352 353%! atom_graphemes(?Atom, ?Graphemes) is det. 354% 355% Relate Atom to a list of its grapheme clusters. Grapheme clusters 356% are "user-perceived characters" as defined by UAX#29 --- e.g. the 357% precomposed U+00E9 (LATIN SMALL LETTER E WITH ACUTE) and the 358% decomposed sequence `e` + U+0301 are both one grapheme, an emoji 359% ZWJ sequence such as `MAN + ZWJ + WOMAN + ZWJ + GIRL` is one 360% grapheme, and a regional-indicator pair (e.g. U+1F1F3 U+1F1F1, 361% rendered as the Dutch flag) is one grapheme. 362% 363% In the forward mode (+Atom, ?Graphemes), Atom is decomposed into 364% a list of atoms, each covering one cluster. In the reverse mode 365% (?Atom, +Graphemes), the elements of Graphemes are concatenated 366% into Atom. Both arguments instantiated means both modes run and 367% the result must agree. 368% 369% ``` 370% ?- atom_codes(A, [0'c, 0'a, 0'f, 0'e, 0x0301]), 371% atom_graphemes(A, Gs). 372% Gs = [c, a, f, G], 373% atom_codes(G, [0'e, 0x0301]). 374% 375% ?- atom_graphemes(A, [a, b, c]). 376% A = abc. 377% ``` 378% 379% @see string_graphemes/2 for the string analogue. 380 381%! string_graphemes(?String, ?Graphemes) is det. 382% 383% As atom_graphemes/2, but the elements of Graphemes are strings. 384 385%! unicode_version(-Version) is det. 386% 387% Version is an atom describing the Unicode version implemented by 388% the linked utf8proc library, e.g. `'15.1.0'`. This drives the 389% normalisation, case-folding and grapheme-cluster predicates in 390% this module, and may differ from the Unicode version of the 391% SWI-Prolog source syntax classifier reported by the read-only 392% Prolog flag `unicode_syntax_version`. 393 394%! unicode_codepoint_valid(+Code) is semidet. 395% 396% True when Code is a non-negative integer that is a valid and 397% _assigned_ Unicode code point. Unassigned code points (general 398% category `Cn`), surrogate halves (`Cs`), and integers outside 399% `0..0x10FFFF` all fail. 400 401 402 /******************************* 403 * SANDBOX * 404 *******************************/ 405 406:- multifile 407 sandbox:safe_primitive/1. 408 409sandbox:safe_primitive(unicode:unicode_property(_,_)). 410sandbox:safe_primitive(unicode:unicode_map(_,_,_)). 411sandbox:safe_primitive(unicode:unicode_property(_,_)). 412sandbox:safe_primitive(unicode:unicode_version(_)). 413sandbox:safe_primitive(unicode:unicode_codepoint_valid(_)). 414sandbox:safe_primitive(unicode:atom_graphemes(_,_)). 415sandbox:safe_primitive(unicode:string_graphemes(_,_))