Utf text

Purpose: Convert UTF (UTF8 or UTF16) string to text.

 utf-text <utf> \
     [ to <text> ] \
     [ length <length> ] \
     [ status <status> ] \
     [ error-text <error text> ]

Copied!

utf-text will convert <utf> text to <text> (specified with "to" clause). If <text> is omitted, then the result of conversion is output.

<utf> is a string that may contain UTF characters (as 2, 3 or 4 bytes representing a unicode character). Encoding creates a string that can be used as a value where text representation of UTF is required. utf-text is performed according to RFC7159 and RFC3629 (UTF standard).

Note that hexadecimal characters used for Unicode (such as \u21d7) are always lowercase. Solidus character ("/") is not escaped, although text-utf will correctly process it if the input has it escaped.

The number of bytes in <utf> to be converted can be specified with <length> in "length" clause. If <length> is not specified, it is the length of string <utf>. Note that a single UTF character can be anywhere between 1 to 4 bytes. For example "љ" is 2 bytes in length.

The status of encoding can be obtained in number <status>. <status> is the string length of the result in <text> (or the number of bytes output if <text> is omitted), or -1 if error occurred (meaning <utf> is an invalid UTF) in which case <text> (if specified) is an empty string and the error text can be obtained in <error text> in "error-text" clause.

Examples

Convert UTF string to text and verify the expected result:

 // UTF string 
 set-string utf_str = "\"Doc\"\n\t\b\f\r\t⇗⇘\t▷◮𝄞ᏫⲠш\n/\"()\t"

 // Convert UTF string to text
 utf-text utf_str status encstatus to text_text

 // This is the text expected
 (( expected_result
 @\"Doc\"\n\t\b\f\r\t\u21d7\u21d8\t\u25b7\u25ee\ud834\udd1e\u13eb\u2ca0\u0448\n/\"()\t
 ))

 // Make sure conversion was okay, decs is the length of the result (encj string)
 if-true text_text equal expected_result and encstatus not-equal -1
     @decode-text worked okay
 end-if

Copied!