Below is a complete, MIT-standard treatment of Unicode-Aware String Processing,
explained from beginner to professional levels, with formal concepts, pitfalls, complete
Java code snippets, correctness invariants, and performance considerations.
This module focuses on correct string processing in the presence of Unicode, not just
ASCII-centric APIs.
MODULE — Unicode-Aware String Processing
(Beginner → Professional, MIT Standard)
0. Why Unicode-Aware Processing Matters
Many string algorithms silently fail when Unicode is involved.
Typical wrong assumptions:
“1 char = 1 character”
“String length = number of characters”
“toLowerCase() is harmless”
“substring() splits characters cleanly”
These assumptions break for:
Accented characters ( é )
Emoji ( 😊)
Indic scripts
Combining marks
Surrogate pairs
LEVEL 1 — Unicode Fundamentals (Beginner)
1. Code Units, Code Points, Characters
Term Meaning
Code unit Storage unit ( char in Java = 16-bit)
Code point Unicode scalar value (U+0000 … U+10FFFF)
Character (grapheme) What a user perceives
Example
java
1/8
String s = " 😊"; // U+1F60A
[Link]([Link]()); // 2 ❌
Why?
Java char is UTF-16
Emoji needs two code units (surrogate pair)
Correct Length (Code Points)
java
int len = [Link](0, [Link]());
[Link](len); // 1 ✅
LEVEL 2 — Iterating Safely Over Unicode (Core Skill)
2.1 ❌ Wrong Way (char-based)
java
for (char c : [Link]()) {
[Link](c);
}
Breaks surrogate pairs.
2.2 ✅ Correct Way (Code Points)
java
[Link]().forEach(cp ->
[Link]([Link](cp))
);
Manual Iteration (MIT-Style Control)
java
for (int i = 0; i < [Link](); ) {
int cp = [Link](i);
[Link](new String([Link](cp)));
i += [Link](cp);
}
Invariant
2/8
At each iteration:
css
i points to the start of a valid Unicode code point
LEVEL 3 — Unicode-Safe Indexing & Substrings
Problem
substring(int, int) operates on code units, not characters.
Example Failure
java
String s = "A 😊B";
String sub = [Link](1, 2); // ❌ breaks surrogate
Unicode-Safe Substring by Code Points
java
static String substringByCodePoints(String s, int start, int end) {
int i = [Link](0, start);
int j = [Link](0, end);
return [Link](i, j);
}
Usage
java
substringByCodePoints("A 😊B", 1, 2); // "😊"
LEVEL 4 — Unicode Normalization (Critical)
4.1 The Problem
Visually identical strings may differ in binary form:
java
String a = "é"; // U+00E9
String b = "e\u0301"; // 'e' + combining accent
[Link](b); // false ❌
3/8
4.2 Unicode Normalization Forms
Form Meaning
NFC Canonical composed
NFD Canonical decomposed
NFKC Compatibility composed
NFKD Compatibility decomposed
Correct Comparison
java
import [Link];
static boolean unicodeEquals(String a, String b) {
return [Link](a, [Link])
.equals(
[Link](b, [Link]));
}
MIT Insight
All Unicode comparisons must define a normalization policy.
LEVEL 5 — Unicode-Aware Case Folding
❌ Locale-Dependent Bug
java
"İ".toLowerCase(); // Wrong in Turkish locale
✅ Correct, Deterministic Case Folding
java
static String foldCase(String s) {
return [Link]([Link]);
}
Why [Link]?
Prevents linguistic side-effects
4/8
Required for protocol-level logic
LEVEL 6 — Unicode-Aware Searching
6.1 Code-Point-Based Search
java
static int indexOfCodePoint(String s, int target) {
for (int i = 0; i < [Link](); ) {
int cp = [Link](i);
if (cp == target) return i;
i += [Link](cp);
}
return -1;
}
6.2 Unicode-Safe Pattern Matching
Before applying KMP / Z / Hashing:
java
String text = [Link](T, [Link]);
String pattern = [Link](P, [Link]);
Then apply algorithms on code points, not chars.
Example: KMP on Code Points
java
static int[] toCodePoints(String s) {
return [Link]().toArray();
}
Then run KMP on int[] instead of char[] .
LEVEL 7 — Grapheme Clusters (Advanced)
Problem
Even code points ≠ user-perceived characters.
Example:
5/8
👩❤️💋👨
→ multiple code points, one grapheme.
Java Limitation
Java does not natively support grapheme segmentation.
ICU4J (Professional Standard)
java
BreakIterator it =
[Link]([Link]);
[Link](s);
for (int start = [Link](), end = [Link]();
end != [Link];
start = end, end = [Link]()) {
[Link]([Link](start, end));
}
MIT Insight
Grapheme-aware processing is mandatory for UI,
optional for algorithms, never free.
LEVEL 8 — Unicode-Safe Sorting & Comparison
Collation vs Binary Order
Binary comparison ≠ linguistic order.
Locale-Aware Collation
java
Collator collator = [Link]([Link]);
[Link]([Link]);
List<String> list = [Link]("é", "e");
[Link](collator);
When to Use
6/8
Context Use Collator
UI sorting ✅
Protocols ❌
Algorithms ❌
Dictionaries ✅
LEVEL 9 — Performance & Memory Considerations
Trade-Offs
Approach Correctness Speed
char-based ❌ Fast
code-point-based ✅ Slower
grapheme-based ✅✅ Slowest
MIT Rule of Thumb
Algorithms → code points
Storage → UTF-16
UI → grapheme clusters
Security → normalize + fold
LEVEL 10 — Security Pitfalls (Professional)
Homoglyph Attacks
text
[Link]
pа[Link] // Cyrillic 'a'
Mitigation
java
String safe = [Link](input, [Link]);
And restrict allowed scripts if needed.
Formal Invariants Summary (MIT Level)
7/8
Code point iteration invariant → no surrogate splitting
Normalization invariant → canonical equivalence
Case folding invariant → locale independence
Substring invariant → valid UTF-16 boundaries
Security invariant → normalization before comparison
Common Exam / Interview Pitfalls
1. Using char as character
2. Ignoring normalization
3. Locale-dependent case logic
4. Breaking surrogate pairs
5. Confusing code points with graphemes
One-Page Recall Summary
Unicode ≠ ASCII
Java char ≠ character
Use code points for algorithms
Normalize before compare
Fold case with [Link]
Grapheme clusters matter for UI
Correctness beats speed in Unicode
Self-Check (MIT Style)
Answer precisely:
1. Why does [Link]() not equal character count?
2. Why must normalization precede comparison?
3. Why is [Link] mandatory for logic?
4. Why are surrogate pairs dangerous?
5. When are grapheme clusters required?
Next Advanced Modules You Can Request
Unicode-aware KMP & Z algorithms
Security issues in string processing
Internationalized search engines
Text processing in NLP pipelines
MIT-style problem sets with proofs
Tell me the next topic you want to continue.
8/8