Rust: baby step — Unicode with Vietnamese text.

We’re looking at some Rust Unicode functionalities using Vietnamese text as examples.

The Slice Type section of the “the book” states:

Note: String slice range indices must occur at valid UTF-8 character boundaries. If you attempt to create a string slice in the middle of a multibyte character, your program will exit with an error. For the purposes of introducing string slices, we are assuming ASCII only in this section; a more thorough discussion of UTF-8 handling is in the “Storing UTF-8 Encoded Text with Strings” section of Chapter 8.

This note is best illustrated with the following Vietnamese poem verse: 

Content of src\example_01.rs:
fn main() {
    let vstr = String::from("Đầu bút nghiễn hề sự cung đao");
    let slice = &vstr[0..3];

    println!("string = [{}]", vstr);
    println!("slice = [{}]", slice);
}
(Đầu bút nghiễn hề sự cung đao means The young husband puts aside his pen and ink, and picks up his sword and long bow to defense his country, from the 18th century poem Chinh Phụ Ngâm — The Ballad Of A Soldier’s Wife.)

Compile and run with the following commands: 

F:\rust\strings>rustc src\example_01.rs
F:\rust\strings>example_01.exe

And as per documentation, the executable exits with an error: 

thread 'main' panicked at 'byte index 3 is not a char boundary; it is inside 'ầ' (bytes 2..5) of `Đầu bút nghiễn hề sự cung đao`', src\example_01.rs:3:18
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Đ is 2 (two) bytes, and ầ is 3 (three) bytes. This explains what the above error is about.

— For me, the obvious question is, for Unicode strings, how, then, do we know the correct byte index to use? We could certain iterate over the characters and calculate the byte index, but that seems overly complicated for such a simple task?

We can use the String‘s chars() iterator to iterate over each character, and print out each character code and length in bytes as follows: 

Content of src\example_02.rs:
fn main() {
    let vstr = String::from("Đầu bút nghiễn hề sự cung đao");

    for char in vstr.chars() {
        println!("char: {}, code: {}, byte size: {}", char, char as u32, char.len_utf8());
    }
}

There’re 29 (twenty nine) characters, the above executable will print out 29 (twenty nine) lines, one for each character.

We can use the String‘s as_bytes() method to iterate over the bytes: 

Content of src\example_03.rs:
fn main() {
    let vstr = String::from("Đầu bút nghiễn hề sự cung đao");

    let bytes = vstr.as_bytes();

    for (i, &item) in bytes.iter().enumerate() {
        println!("i: {}, item: {}", i, item);
    }
}

There’re 40 (forty) bytes in total, the first 5 (five) bytes are: 196, 144, 225, 186 and 167 which correspond to the first two (2) characters Đầ.

Following from_utf8(…), if we feed the above 5 (five) bytes to this method, we’d get Đầ: 

Content of src\example_04.rs:
fn main() {
    let first_two_char_bytes = vec![196, 144, 225, 186, 167];

    let first_two_char = String::from_utf8(first_two_char_bytes).unwrap();

    println!("{}", first_two_char);
}

Letters f, j, w and z are not official Vietnamese alphabets, (most Vietnamese are aware of them, even if they don’t know English), in addition to the remaining 22 (twenty two) letters, there’re another 67 (sixty seven) additional letters, as in English, there’re both upper case and lower case. Some of these letters are not uniquely Vietnamese, they are found in other languages, following are a few of them: à, â, đ, è, é, ê, ì, ò, ô and ù.

We could have global constants for the 67 (sixty seven) letters as follows: 

static VIETNAMESE_UPPERCASE: &str = "ÁÀẢÃẠĂẮẰẲẴẶÂẤẦẨẪẬĐÉÈẺẼẸÊẾỀỂỄỆÍÌỈĨỊÓÒỎÕỌÔỐỒỔỖỘƠỚỜỞỠỢÚÙỦŨỤƯỨỪỬỮỰÝỲỶỸỴ";
static VIETNAMESE_LOWERCASE: &str = "áàảãạăắằẳẵặâấầẩẫậđéèẻẽẹêếềểễệíìỉĩịóòỏõọôốồổỗộơớờởỡợúùủũụưứừửữựýỳỷỹỵ";

They’re listed based on two orders: Latin alphabets, then Vietnamese diacritic tonal marks, that is, the acute (e.g. Á, Ố etc.) tonal mark comes before any other marks.

For the English alphabets, the ASCII codes are in sequence. That is, the ASCII code for capital A is 65, B is 66 and so on. This is not the case for the “Vietnamese” letters as seen in the following table:

Upper CaseLower Case
CharCodeByte #.CharCodeByte #.
Á1932á2252
À1922à2242
Ả78423ả78433
Ã1952ã2272
Ạ78403ạ78413
Ă2582ă2592
Ắ78543ắ78553
Ằ78563ằ78573
Ẳ78583ẳ78593
Ẵ78603ẵ78613
Ặ78623ặ78633
Â1942â2262
Ấ78443ấ78453
Ầ78463ầ78473
Ẩ78483ẩ78493
Ẫ78503ẫ78513
Ậ78523ậ78533
Đ2722đ2732
É2012é2332
È2002è2322
Ẻ78663ẻ78673
Ẽ78683ẽ78693
Ẹ78643ẹ78653
Ê2022ê2342
Ế78703ế78713
Ề78723ề78733
Ể78743ể78753
Ễ78763ễ78773
Ệ78783ệ78793
Í2052í2372
Ì2042ì2362
Ỉ78803ỉ78813
Ĩ2962ĩ2972
Ị78823ị78833
Ó2112ó2432
Ò2102ò2422
Ỏ78863ỏ78873
Õ2132õ2452
Ọ78843ọ78853
Ô2122ô2442
Ố78883ố78893
Ồ78903ồ78913
Ổ78923ổ78933
Ỗ78943ỗ78953
Ộ78963ộ78973
Ơ4162ơ4172
Ớ78983ớ78993
Ờ79003ờ79013
Ở79023ở79033
Ỡ79043ỡ79053
Ợ79063ợ79073
Ú2182ú2502
Ù2172ù2492
Ủ79103ủ79113
Ũ3602ũ3612
Ụ79083ụ79093
Ư4312ư4322
Ứ79123ứ79133
Ừ79143ừ79153
Ử79163ử79173
Ữ79183ữ79193
Ự79203ự79213
Ý2212ý2532
Ỳ79223ỳ79233
Ỷ79263ỷ79273
Ỹ79283ỹ79293
Ỵ79243ỵ79253
Vietnamese specific alphabets: character codes and byte sizes.

I think because the extended ASCII table already includes a few, and so the Unicode Consortium just allocates new codes for the missing ones.

(Back in the 1990s, to display Vietnamese, some of the not often used displayable extended ASCII characters were redrawn to look like Vietnamese letters, and the keyboard was programmed to match. For example, the earliest convention is VNI, originated from the United States, whereby u followed by ? produces ủ which replaces u. This convention is still in used today, but with Unicode.)

Rust provides several methods for case conversions: to_lowercase(…), to_uppercase(…), to_ascii_lowercase(…), to_ascii_uppercase(…), make_ascii_lowercase(…) and make_ascii_uppercase(…).

The first two methods work with Unicode strings: 

Content of src\example_05.rs:
static VIETNAMESE_UPPERCASE: &str = "ÁÀẢÃẠĂẮẰẲẴẶÂẤẦẨẪẬĐÉÈẺẼẸÊẾỀỂỄỆÍÌỈĨỊÓÒỎÕỌÔỐỒỔỖỘƠỚỜỞỠỢÚÙỦŨỤƯỨỪỬỮỰÝỲỶỸỴ";
static VIETNAMESE_LOWERCASE: &str = "áàảãạăắằẳẵặâấầẩẫậđéèẻẽẹêếềểễệíìỉĩịóòỏõọôốồổỗộơớờởỡợúùủũụưứừửữựýỳỷỹỵ";

fn main() {
    let s = String::from(VIETNAMESE_UPPERCASE.to_lowercase());
    assert_eq!(s, VIETNAMESE_LOWERCASE);

    // Does not work.
    let s = String::from(VIETNAMESE_UPPERCASE.to_ascii_lowercase());
    assert_eq!(s, VIETNAMESE_UPPERCASE);

    // Does not work.
    let mut s = String::from(VIETNAMESE_LOWERCASE);
    s.make_ascii_uppercase();
    assert_eq!(s, VIETNAMESE_LOWERCASE);
}

I’m guessing that when we’re certain we only work with ASCII strings, it’s better to call ASCII-based methods?

I’m not sure how relevant this post is to other people, I tried to understand this character versus byte issue in Rust, and these’re the example codes which I’ve written to understand the issue, I document it so that I have a reference to go back to should the need arise.

I hope you find this post relevant somehow… Thank you for reading and stay safe as always.

✿✿✿

Feature image source:

Leave a comment

Design a site like this with WordPress.com
Get started