Unicode and UTF-8: Why Your Emoji Breaks the Database

The first time this bug bit me, a product review in the database just stopped. The user had typed “works great 👍” and the stored row ended after the space. No exception, no alert, nothing in the logs. The browser was fine. The application code was fine. The string had lost its last character somewhere between the form and the disk, and it took me an embarrassing afternoon to work out where.

That bug and about six of its cousins all come from the same confusion. Three different things get called “text” and get treated as one thing. A character is an abstract idea, the notion of a thumbs up or a lowercase e with an accent on it. A code point is the number Unicode assigns to that idea. An encoding is the rule that turns that number into bytes. Separate those three in your head and this whole class of bug becomes boring, which is the highest compliment you can pay a bug.

Three things that are not the same thing

Unicode is a catalog. It says: there is a character called THUMBS UP SIGN, and its number is 1F44D in hexadecimal, written U+1F44D. Version 17.0 of the standard, published on September 9, 2025, catalogs 159,801 characters, 4,803 of them new in that release.^[2] The catalog reserves room for 1,114,112 code points in total, from U+0000 up to U+10FFFF.^[1]

Notice what the catalog does not say. It does not say how big the number is on disk. That is the encoder's job, and there is more than one encoder.

The reason this matters practically is that different parts of your stack pick different encoders and none of them tell you. The browser sends UTF-8. Your JavaScript runtime holds UTF-16 in memory. Your database column has its own character set. Your log aggregator has an opinion too. Text survives that trip only if every hop agrees, which is also the argument for pinning the encoding explicitly at every API boundary you own rather than hoping the default is the one you wanted.

How UTF-8 turns a number into bytes

UTF-8 was specified in its current form by RFC 3629 in November 2003. It is variable length: one to four octets per character, covering the whole U+0000 to U+10FFFF range.^[1] The scheme is worth knowing at the bit level, because once you have seen it you can read a corrupted byte stream by eye.

Code point range        UTF-8 bytes (x = payload bits)

U+0000  .. U+007F       0xxxxxxx
U+0080  .. U+07FF       110xxxxx  10xxxxxx
U+0800  .. U+FFFF       1110xxxx  10xxxxxx  10xxxxxx
U+10000 .. U+10FFFF     11110xxx  10xxxxxx  10xxxxxx  10xxxxxx

Read the first byte and you know everything. A byte starting with 0 is a one-byte character, and the remaining seven bits are the code point, which is exactly ASCII. That is the trick that made UTF-8 win: every ASCII file ever written is already a valid UTF-8 file, byte for byte, no conversion. A byte starting 110 means “two bytes, I am the first.” 1110 means three. 11110 means four. And every continuation byte starts with 10, a pattern no leading byte ever uses.

That last detail is the elegant part. Because leading and continuation bytes are distinguishable on sight, you can drop into the middle of a UTF-8 stream, scan backwards past any byte starting with 10, and you have found a character boundary. The RFC calls this out directly: character boundaries are easily found from anywhere in an octet stream.^[1] A truncated file loses one character, not the rest of the document.

U+1F44D  (THUMBS UP SIGN)  =  1 1111 0100 0100 1101

pad to 21 payload bits:      000  011111  010001  001101
                              |      |       |       |
                          11110000 10011111 10010001 10001101
                            0xF0     0x9F     0x91     0x8D

How long is this string? Pick a number, any number

Here is the part that took me longest to internalize, and the reason these bugs keep finding new places to live. There is no single correct answer to “how long is this string.” There are at least four answers, all correct, and the bug is always a layer that picked a different one than you did.

Add the other two units and it gets worse. JavaScript strings are sequences of 16-bit UTF-16 code units, so anything above U+FFFF is stored as a surrogate pair and counted twice.^[5] Java made the same choice in the 1990s and is stuck with it. Python 3 counts code points, which is one of the quieter reasons text handling feels different when you are coming to Python from Java.

"👍".length            // 2   <- UTF-16 code units
[..."👍"].length       // 1   <- code points
"👍".charCodeAt(0)     // 55357 = 0xD83D, a lone high surrogate

"👩‍👩‍👦".length          // 8
[..."👩‍👩‍👦"].length     // 5   <- three people, two zero-width joiners

And the fourth answer, the one your user actually means: the grapheme cluster, which Unicode Standard Annex #29 defines as the unit a reader thinks of as a single character.^[4] The family emoji is one of those. So is a Devanagari syllable with its vowel sign. If you are truncating a display name, that is the unit you want, and neither JavaScript nor Python gives it to you without a library.

MySQL's utf8 was never UTF-8

Back to the truncated review. MySQL has a character set called utf8. It is not UTF-8. It is a deprecated alias for utf8mb3, which stores a maximum of three bytes per character and supports only the Basic Multilingual Plane, the first 65,536 code points.^[3] Everything above U+FFFF, which is every emoji, a lot of historic scripts, and the newer CJK extension blocks, needs a fourth byte that the column simply does not have.

-- column is CHARACTER SET utf8mb3, strict mode on (the default)
INSERT INTO reviews (body) VALUES ('works great 👍');
ERROR 1366 (HY000): Incorrect string value: '\xF0\x9F\x91\x8D'
                    for column 'body' at row 1

-- same insert with strict mode off: no error, and the row reads
-- 'works great ' with the emoji and everything after it gone

The failure mode depends entirely on your SQL mode, and this is the detail that decides whether you find the bug in ten minutes or ten months. Strict mode is in MySQL's default set today, and it turns the bad insert into an error you cannot miss.^[6] Without strict mode, MySQL inserts an adjusted value and produces a warning instead,^[6] which in practice means the row is written, truncated at the offending character, and your application never asks about warnings. That is the version I hit. Silent data loss, one column at a time.

Fixing it is four changes, not one: the column, the table default, the database default, and the client connection charset. Miss the connection and a correctly configured utf8mb4 column still receives mangled bytes. Then read the migration page before you run the ALTER, because index limits move with the byte width. On InnoDB tables with COMPACT or REDUNDANT row format the index prefix limit is 767 bytes, so an indexed column drops from 255 characters under utf8mb3 to 191 under utf8mb4; COMPRESSED and DYNAMIC formats allow 3,072 bytes, which is 1,024 characters against 768.^[7] The number 191 in a legacy schema is a fossil of exactly this migration.

None of this is a reason to run from relational databases, and if you are weighing that trade there are better axes to argue about in the SQL versus NoSQL comparison. A document store with a bad connection charset corrupts text just as cheerfully. The lesson is narrower: a field named utf8 lied for over a decade, and the only defense is checking what a name actually means instead of what it sounds like.

Mojibake and normalization, the two that survive the migration

Two more failures, and neither is fixed by switching to utf8mb4.

The first is mojibake, the Japanese term for text turned to garbage by being decoded with the wrong character set. UTF-8 bytes read as Latin-1 is the classic. The word café ends in the bytes C3 A9, and Latin-1 sees two separate one-byte characters there, so you get café. The encouraging part is that the bytes are usually intact and the mistake is only in the decode step, so it is reversible if you catch it before something re-encodes and saves the mangled version.

The second is normalization, and it is sneakier because nothing looks broken. Unicode gives you two ways to write é. One is the single code point U+00E9. The other is a plain e, U+0065, followed by a combining acute accent, U+0301. They render identically. They are not equal. Unicode Standard Annex #15 defines the four normalization forms that resolve this, and the one you want almost always is NFC, canonical decomposition followed by canonical composition, which collapses the pair back into the single code point.^[8]

>>> from unicodedata import normalize
>>> nfc, nfd = normalize("NFC", "café"), normalize("NFD", "café")
>>> nfc == nfd
False
>>> len(nfc), len(nfd)
(4, 5)
>>> nfc[-1].encode().hex(" ")      # composed
'c3 a9'
>>> nfd[-2:].encode().hex(" ")     # e + combining acute
'65 cc 81'

Where this bites: a user pastes a name from a Mac, which hands you the decomposed form, and your uniqueness check on it passes because the stored copy is composed. Now you have two accounts with visually identical names. Or a search box that never matches a document because one side was normalized and the other was not. The bug report says “search is broken for accented names” and nothing in your query looks wrong.

The four rules that end this class of bug

None of this is hard. It is just unforgiving, because every layer gets a vote and silence reads as agreement.

1. UTF-8 everywhere, declared explicitly. Database, connection, HTTP headers, file reads, the lot. Never rely on a platform default, because the default differs by platform and moves between versions.
2. utf8mb4 in MySQL, always. Column, table, database and client. Treat any surviving utf8 or utf8mb3 in a schema as a scheduled outage.
3. Normalize to NFC at the boundary. One line, at the point text enters the system, before storage or comparison.
4. Say which unit you are counting. Bytes for storage limits, code points for validation, grapheme clusters for anything a human sees. A length limit with no stated unit is a defect, not a decision.

Learn the four bytes of one emoji properly and you never lose an afternoon to this again. F0 9F 91 8D. Go check what your reviews table is set to.