=head1 NAME perlunicode - Unicode support in Perl =head1 DESCRIPTION If you haven't already, before reading this document, you should become familiar with both L and L. Unicode aims to B-fy the en-B-ings of all the world's character sets into a single Standard. For quite a few of the various coding standards that existed when Unicode was first created, converting from each to Unicode essentially meant adding a constant to each code point in the original standard, and converting back meant just subtracting that same constant. For ASCII and ISO-8859-1, the constant is 0. For ISO-8859-5, (Cyrillic) the constant is 864; for Hebrew (ISO-8859-8), it's 1488; Thai (ISO-8859-11), 3424; and so forth. This made it easy to do the conversions, and facilitated the adoption of Unicode. And it worked; nowadays, those legacy standards are rarely used. Most everyone uses Unicode. Unicode is a comprehensive standard. It specifies many things outside the scope of Perl, such as how to display sequences of characters. For a full discussion of all aspects of Unicode, see L. =head2 Important Caveats Even though some of this section may not be understandable to you on first reading, we think it's important enough to highlight some of the gotchas before delving further, so here goes: Unicode support is an extensive requirement. While Perl does not implement the Unicode standard or the accompanying technical reports from cover to cover, Perl does support many Unicode features. Also, the use of Unicode may present security issues that aren't obvious, see L below. =over 4 =item Safest if you C In order to preserve backward compatibility, Perl does not turn on full internal Unicode support unless the pragma L>|feature/The 'unicode_strings' feature> is specified. (This is automatically selected if you S> or higher.) Failure to do this can trigger unexpected surprises. See L below. This pragma doesn't affect I/O. Nor does it change the internal representation of strings, only their interpretation. There are still several places where Unicode isn't fully supported, such as in filenames. =item Input and Output Layers Use the C<:encoding(...)> layer to read from and write to filehandles using the specified encoding. (See L.) =item You must convert your non-ASCII, non-UTF-8 Perl scripts to be UTF-8. The L module has been deprecated since perl 5.18 and the perl internals it requires have been removed with perl 5.26. =item C still needed to enable L in scripts If your Perl script is itself encoded in L, the S> pragma must be explicitly included to enable recognition of that (in string or regular expression literals, or in identifier names). B> is needed.> (See L). If a Perl script begins with the bytes that form the UTF-8 encoding of the Unicode BYTE ORDER MARK (C, see L), those bytes are completely ignored. =item L scripts autodetected If a Perl script begins with the Unicode C (UTF-16LE, UTF16-BE), or if the script looks like non-C-marked UTF-16 of either endianness, Perl will correctly read in the script as the appropriate Unicode encoding. =back =head2 Byte and Character Semantics Before Unicode, most encodings used 8 bits (a single byte) to encode each character. Thus a character was a byte, and a byte was a character, and there could be only 256 or fewer possible characters. "Byte Semantics" in the title of this section refers to this behavior. There was no need to distinguish between "Byte" and "Character". Then along comes Unicode which has room for over a million characters (and Perl allows for even more). This means that a character may require more than a single byte to represent it, and so the two terms are no longer equivalent. What matter are the characters as whole entities, and not usually the bytes that comprise them. That's what the term "Character Semantics" in the title of this section refers to. Perl had to change internally to decouple "bytes" from "characters". It is important that you too change your ideas, if you haven't already, so that "byte" and "character" no longer mean the same thing in your mind. The basic building block of Perl strings has always been a "character". The changes basically come down to that the implementation no longer thinks that a character is always just a single byte. There are various things to note: =over 4 =item * String handling functions, for the most part, continue to operate in terms of characters. C, for example, returns the number of characters in a string, just as before. But that number no longer is necessarily the same as the number of bytes in the string (there may be more bytes than characters). The other such functions include C, C, C, C, C, C, C, C, and C. The exceptions are: =over 4 =item * the bit-oriented C E =item * the byte-oriented C/C C<"C"> format However, the C specifier does operate on whole characters, as does the C specifier. =item * some operators that interact with the platform's operating system Operators dealing with filenames are examples. =item * when the functions are called from within the scope of the S>> pragma Likely, you should use this only for debugging anyway. =back =item * Strings--including hash keys--and regular expression patterns may contain characters that have ordinal values larger than 255. If you use a Unicode editor to edit your program, Unicode characters may occur directly within the literal strings in UTF-8 encoding, or UTF-16. (The former requires a C, the latter may require a C.) L gives other ways to place non-ASCII characters in your strings. =item * The C and C functions work on whole characters. =item * Regular expressions match whole characters. For example, C<"."> matches a whole character instead of only a single byte. =item * The C operator translates whole characters. (Note that the C functionality has been removed. For similar functionality to that, see C and C). =item * C reverses by character rather than by byte. =item * The bit string operators, C<& | ^ ~> and (starting in v5.22) C<&. |. ^. ~.> can operate on bit strings encoded in UTF-8, but this can give unexpected results if any of the strings contain code points above 0xFF. Starting in v5.28, it is a fatal error to have such an operand. Otherwise, the operation is performed on a non-UTF-8 copy of the operand. If you're not sure about the encoding of a string, downgrade it before using any of these operators; you can use L|utf8/Utility functions>. =back The bottom line is that Perl has always practiced "Character Semantics", but with the advent of Unicode, that is now different than "Byte Semantics". =head2 ASCII Rules versus Unicode Rules Before Unicode, when a character was a byte was a character, Perl knew only about the 128 characters defined by ASCII, code points 0 through 127 (except for under L>|perllocale>). That left the code points 128 to 255 as unassigned, and available for whatever use a program might want. The only semantics they have is their ordinal numbers, and that they are members of none of the non-negative character classes. None are considered to match C<\w> for example, but all match C<\W>. Unicode, of course, assigns each of those code points a particular meaning (along with ones above 255). To preserve backward compatibility, Perl only uses the Unicode meanings when there is some indication that Unicode is what is intended; otherwise the non-ASCII code points remain treated as if they are unassigned. Here are the ways that Perl knows that a string should be treated as Unicode: =over =item * Within the scope of S> If the whole program is Unicode (signified by using 8-bit Bnicode Bransformation Bormat), then all literal strings within it must be Unicode. =item * Within the scope of L>|feature/The 'unicode_strings' feature> This pragma was created so you can explicitly tell Perl that operations executed within its scope are to use Unicode rules. More operations are affected with newer perls. See L. =item * Within the scope of S> or higher This implicitly turns on S>. =item * Within the scope of L>|perllocale/Unicode and UTF-8>, or L>|perllocale> and the current locale is a UTF-8 locale. The former is defined to imply Unicode handling; and the latter indicates a Unicode locale, hence a Unicode interpretation of all strings within it. =item * When the string contains a Unicode-only code point Perl has never accepted code points above 255 without them being Unicode, so their use implies Unicode for the whole string. =item * When the string contains a Unicode named code point C<\N{...}> The C<\N{...}> construct explicitly refers to a Unicode code point, even if it is one that is also in ASCII. Therefore the string containing it must be Unicode. =item * When the string has come from an external source marked as Unicode The L|perlrun/-C [numberElist]> command line option can specify that certain inputs to the program are Unicode, and the values of this can be read by your Perl code, see L. =item * When the string has been upgraded to UTF-8 The function L|utf8/Utility functions> can be explicitly used to permanently (unless a subsequent C is called) cause a string to be treated as Unicode. =item * There are additional methods for regular expression patterns A pattern that is compiled with the C<< /u >> or C<< /a >> modifiers is treated as Unicode (though there are some restrictions with C<< /a >>). Under the C<< /d >> and C<< /l >> modifiers, there are several other indications for Unicode; see L. =back Note that all of the above are overridden within the scope of C>; but you should be using this pragma only for debugging. Note also that some interactions with the platform's operating system never use Unicode rules. When Unicode rules are in effect: =over 4 =item * Case translation operators use the Unicode case translation tables. Note that C, or C<\U> in interpolated strings, translates to uppercase, while C, or C<\u> in interpolated strings, translates to titlecase in languages that make the distinction (which is equivalent to uppercase in languages without the distinction). There is a CPAN module, C>, which allows you to define your own mappings to be used in C, C, C, C, and C (or their double-quoted string inlined versions such as C<\U>). (Prior to Perl 5.16, this functionality was partially provided in the Perl core, but suffered from a number of insurmountable drawbacks, so the CPAN module was written instead.) =item * Character classes in regular expressions match based on the character properties specified in the Unicode properties database. C<\w> can be used to match a Japanese ideograph, for instance; and C<[[:digit:]]> a Bengali number. =item * Named Unicode properties, scripts, and block ranges may be used (like bracketed character classes) by using the C<\p{}> "matches property" construct and the C<\P{}> negation, "doesn't match property". See L for more details. You can define your own character properties and use them in the regular expression with the C<\p{}> or C<\P{}> construct. See L for more details. =back =head2 Extended Grapheme Clusters (Logical characters) Consider a character, say C. It could appear with various marks around it, such as an acute accent, or a circumflex, or various hooks, circles, arrows, I, above, below, to one side or the other, I. There are many possibilities among the world's languages. The number of combinations is astronomical, and if there were a character for each combination, it would soon exhaust Unicode's more than a million possible characters. So Unicode took a different approach: there is a character for the base C, and a character for each of the possible marks, and these can be variously combined to get a final logical character. So a logical character--what appears to be a single character--can be a sequence of more than one individual characters. The Unicode standard calls these "extended grapheme clusters" (which is an improved version of the no-longer much used "grapheme cluster"); Perl furnishes the C<\X> regular expression construct to match such sequences in their entirety. But Unicode's intent is to unify the existing character set standards and practices, and several pre-existing standards have single characters that mean the same thing as some of these combinations, like ISO-8859-1, which has quite a few of them. For example, C<"LATIN CAPITAL LETTER E WITH ACUTE"> was already in this standard when Unicode came along. Unicode therefore added it to its repertoire as that single character. But this character is considered by Unicode to be equivalent to the sequence consisting of the character C<"LATIN CAPITAL LETTER E"> followed by the character C<"COMBINING ACUTE ACCENT">. C<"LATIN CAPITAL LETTER E WITH ACUTE"> is called a "pre-composed" character, and its equivalence with the "E" and the "COMBINING ACCENT" sequence is called canonical equivalence. All pre-composed characters are said to have a decomposition (into the equivalent sequence), and the decomposition type is also called canonical. A string may be comprised as much as possible of precomposed characters, or it may be comprised of entirely decomposed characters. Unicode calls these respectively, "Normalization Form Composed" (NFC) and "Normalization Form Decomposed". The C> module contains functions that convert between the two. A string may also have both composed characters and decomposed characters; this module can be used to make it all one or the other. You may be presented with strings in any of these equivalent forms. There is currently nothing in Perl 5 that ignores the differences. So you'll have to specially handle it. The usual advice is to convert your inputs to C before processing further. For more detailed information, see L. =head2 Unicode Character Properties (The only time that Perl considers a sequence of individual code points as a single logical character is in the C<\X> construct, already mentioned above. Therefore "character" in this discussion means a single Unicode code point.) Very nearly all Unicode character properties are accessible through regular expressions by using the C<\p{}> "matches property" construct and the C<\P{}> "doesn't match property" for its negation. For instance, C<\p{Uppercase}> matches any single character with the Unicode C<"Uppercase"> property, while C<\p{L}> matches any character with a C of C<"L"> (letter) property (see L below). Brackets are not required for single letter property names, so C<\p{L}> is equivalent to C<\pL>. More formally, C<\p{Uppercase}> matches any single character whose Unicode C property value is C, and C<\P{Uppercase}> matches any character whose C property value is C, and they could have been written as C<\p{Uppercase=True}> and C<\p{Uppercase=False}>, respectively. This formality is needed when properties are not binary; that is, if they can take on more values than just C and C. For example, the C property (see L below), can take on several different values, such as C, C, C, and others. To match these, one needs to specify both the property name (C), AND the value being matched against (C, C, I). This is done, as in the examples above, by having the two components separated by an equal sign (or interchangeably, a colon), like C<\p{Bidi_Class: Left}>. All Unicode-defined character properties may be written in these compound forms of C<\p{I=I}> or C<\p{I:I}>, but Perl provides some additional properties that are written only in the single form, as well as single-form short-cuts for all binary properties and certain others described below, in which you may omit the property name and the equals or colon separator. Most Unicode character properties have at least two synonyms (or aliases if you prefer): a short one that is easier to type and a longer one that is more descriptive and hence easier to understand. Thus the C<"L"> and C<"Letter"> properties above are equivalent and can be used interchangeably. Likewise, C<"Upper"> is a synonym for C<"Uppercase">, and we could have written C<\p{Uppercase}> equivalently as C<\p{Upper}>. Also, there are typically various synonyms for the values the property can be. For binary properties, C<"True"> has 3 synonyms: C<"T">, C<"Yes">, and C<"Y">; and C<"False"> has correspondingly C<"F">, C<"No">, and C<"N">. But be careful. A short form of a value for one property may not mean the same thing as the short form spelled the same for another. Thus, for the C> property, C<"L"> means C<"Letter">, but for the L|/Bidirectional Character Types> property, C<"L"> means C<"Left">. A complete list of properties and synonyms is in L. Upper/lower case differences in property names and values are irrelevant; thus C<\p{Upper}> means the same thing as C<\p{upper}> or even C<\p{UpPeR}>. Similarly, you can add or subtract underscores anywhere in the middle of a word, so that these are also equivalent to C<\p{U_p_p_e_r}>. And white space is generally irrelevant adjacent to non-word characters, such as the braces and the equals or colon separators, so C<\p{ Upper }> and C<\p{ Upper_case : Y }> are equivalent to these as well. In fact, white space and even hyphens can usually be added or deleted anywhere. So even C<\p{ Up-per case = Yes}> is equivalent. All this is called "loose-matching" by Unicode. The "name" property has some restrictions on this due to a few outlier names. Full details are given in L. The few places where stricter matching is used is in the middle of numbers, the "name" property, and in the Perl extension properties that begin or end with an underscore. Stricter matching cares about white space (except adjacent to non-word characters), hyphens, and non-interior underscores. You can also use negation in both C<\p{}> and C<\P{}> by introducing a caret (C<^>) between the first brace and the property name: C<\p{^Tamil}> is equal to C<\P{Tamil}>. Almost all properties are immune to case-insensitive matching. That is, adding a C regular expression modifier does not change what they match. There are two sets that are affected. The first set is C, C, and C, all of which match C under C matching. And the second set is C, C, and C, all of which match C under C matching. This set also includes its subsets C and C both of which under C match C. (The difference between these sets is that some things, such as Roman numerals, come in both upper and lower case so they are C, but aren't considered letters, so they aren't C's.) See L for special considerations when matching Unicode properties against non-Unicode code points. =head3 B Every Unicode character is assigned a general category, which is the "most usual categorization of a character" (from L). The compound way of writing these is like C<\p{General_Category=Number}> (short: C<\p{gc:n}>). But Perl furnishes shortcuts in which everything up through the equal or colon separator is omitted. So you can instead just write C<\pN>. Here are the short and long forms of the values the C property can have: Short Long L Letter LC, L& Cased_Letter (that is: [\p{Ll}\p{Lu}\p{Lt}]) Lu Uppercase_Letter Ll Lowercase_Letter Lt Titlecase_Letter Lm Modifier_Letter Lo Other_Letter M Mark Mn Nonspacing_Mark Mc Spacing_Mark Me Enclosing_Mark N Number Nd Decimal_Number (also Digit) Nl Letter_Number No Other_Number P Punctuation (also Punct) Pc Connector_Punctuation Pd Dash_Punctuation Ps Open_Punctuation Pe Close_Punctuation Pi Initial_Punctuation (may behave like Ps or Pe depending on usage) Pf Final_Punctuation (may behave like Ps or Pe depending on usage) Po Other_Punctuation S Symbol Sm Math_Symbol Sc Currency_Symbol Sk Modifier_Symbol So Other_Symbol Z Separator Zs Space_Separator Zl Line_Separator Zp Paragraph_Separator C Other Cc Control (also Cntrl) Cf Format Cs Surrogate Co Private_Use Cn Unassigned Single-letter properties match all characters in any of the two-letter sub-properties starting with the same letter. C and C are special: both are aliases for the set consisting of everything matched by C, C, and C. =head3 B Because scripts differ in their directionality (Hebrew and Arabic are written right to left, for example) Unicode supplies a C property. Some of the values this property can have are: Value Meaning L Left-to-Right LRE Left-to-Right Embedding LRO Left-to-Right Override R Right-to-Left AL Arabic Letter RLE Right-to-Left Embedding RLO Right-to-Left Override PDF Pop Directional Format EN European Number ES European Separator ET European Terminator AN Arabic Number CS Common Separator NSM Non-Spacing Mark BN Boundary Neutral B Paragraph Separator S Segment Separator WS Whitespace ON Other Neutrals This property is always written in the compound form. For example, C<\p{Bidi_Class:R}> matches characters that are normally written right to left. Unlike the C> property, this property can have more values added in a future Unicode release. Those listed above comprised the complete set for many Unicode releases, but others were added in Unicode 6.3; you can always find what the current ones are in L. And L describes how to use them. =head3 B The world's languages are written in many different scripts. This sentence (unless you're reading it in translation) is written in Latin, while Russian is written in Cyrillic, and Greek is written in, well, Greek; Japanese mainly in Hiragana or Katakana. There are many more. The Unicode C