Most Of The Time People Encode

Most of the time people encode something, they don't realize they're doing it.

You hit "save" on a text file. You copy a weird symbol from a website and paste it into an email. You upload a video to YouTube. All of it — encoding. On the flip side, you scan a QR code at a restaurant. Invisible, automatic, and usually fine.

Until it isn't.

Then you get mojibake. Or a CSV that turns your carefully typed Japanese characters into question marks. Or a video that plays audio but shows green blocks. And suddenly you care a lot about encoding.

Here's the thing: encoding isn't one thing. Worth adding: it's a family of problems that all share the same root — translating human stuff into machine stuff, and back again. Most guides treat them separately. But in practice? They bleed into each other. A developer debugging a database collation issue is fighting the same battle as a creator wondering why their 4K export looks like mud on TikTok.

Let's untangle it.

What Encoding Actually Is

Strip away the jargon and encoding is just a agreed-upon mapping between two representations.

You have information in form A (letters, light, sound, gestures). You need it in form B (bytes, voltage levels, radio waves, pits on a disc). The encoding is the rulebook that says "this maps to that.

That's it. The complexity comes from:

• How many forms A and B exist
• Whether the mapping is lossless or lossy
• Who agreed on the rulebook — and whether everyone actually uses it

Character encoding: the one you swim in daily

Every text file, every database row, every HTTP header, every terminal output — it's all bytes until something applies a character encoding to interpret them Simple, but easy to overlook..

ASCII was the first widespread standard. 128 characters. English letters, numbers, basic punctuation, control codes. Perfect for 1960s America. Useless for literally everyone else.

Then came the chaos years. Big5. Shift-JIS. Windows-1252 (almost Latin-1 but not quite). GB2312. ISO-8859-1 (Latin-1). Think about it: kOI8-R. Dozens of incompatible 8-bit extensions. If you opened a file with the wrong one, you got garbage. People called it "mojibake" — Japanese for "character mutation" — and the name stuck Worth keeping that in mind. Still holds up..

Unicode changed the game. One code point for every character in every writing system. Think about it: emoji included. On top of that, over 149,000 characters as of Unicode 15. Even so, 1. But Unicode itself isn't an encoding — it's a character set. You still need to turn those code points into bytes.

That's where UTF-8, UTF-16, and UTF-32 come in.

UTF-8 won. It's backward compatible with ASCII (the first 128 code points are identical byte-for-byte). It's variable-width: 1 byte for ASCII, 2–3 bytes for most world scripts, 4 bytes for emoji and rare characters. It's self-synchronizing — you can jump into the middle of a byte stream and find the next character boundary. And it's now the default for HTML, XML, JSON, Linux, macOS, and the web at large.

UTF-16 persists in Java, JavaScript (internally), Windows APIs, and some databases. It uses 2 bytes for the Basic Multilingual Plane, 4 bytes (surrogate pairs) for everything else. It's not self-synchronizing. It has endianness issues (UTF-16LE vs UTF-16BE). Avoid it unless you're forced to use it.

UTF-32 uses 4 bytes for everything. Simple, fixed-width, wastes enormous space. Almost never used for storage or transmission.

If you're building something new: **use UTF-8 everywhere.Which means ** Database? UTF-8. On the flip side, aPI? UTF-8. Now, config files? UTF-8. Source code? UTF-8. The only exception is when a legacy system forces your hand — and even then, convert at the boundary Not complicated — just consistent..

Video encoding: the one you argue about

Video encoding is a different beast. Here the mapping isn't character → byte. It's raw pixel data → compressed bitstream That alone is useful..

Raw 1080p30 video: 1920 × 1080 × 30 × 3 bytes (RGB) ≈ 186 MB per second. A two-hour movie? 1.3 TB. Day to day, uncompressed 4K60? Also, you're looking at 6+ TB/hour. Nobody stores or streams that And that's really what it comes down to. Less friction, more output..

So we compress. Which means Lossy compression — throwing away data the human eye/brain won't miss. The art is throwing away the right data.

Modern codecs (H.Worth adding: 264/AVC, H. 265/HEVC, VP9, AV1) all work on similar principles:

1. Block partitioning — split frames into macroblocks (16×16, 64×64, variable)
2. Because of that, Prediction — guess what a block looks like from neighbors (intra) or previous/next frames (inter)
3. Transform — convert prediction error (residual) to frequency domain (DCT/DST)
4. Quantization — divide coefficients by a quantization matrix; this is where loss happens

The bitrate controls quality. But the relationship isn't linear. On top of that, higher bitrate = less quantization = more detail = larger file. 264 at 8 Mbps. 265 at 5 Mbps can look better than H.H.AV1 at 4 Mbps can beat both.

Containers ≠ codecs. This trips people up constantly. .mp4, .mkv, .mov, .webm — these are containers. They hold video streams, audio streams, subtitles, chapters, metadata. The codec is what's inside. An .mp4 can contain H.264, H.265, AV1, MPEG-2, even ProRes. Don't say "I encoded it as MP4." Say "I encoded H.265 in an MP4 container."

Audio encoding: the one you ignore

Same story. Raw CD audio: 44.So 1 kHz × 16-bit × 2 channels = 1. Still, 4 Mbps. Because of that, mP3 at 128 kbps throws away ~90% of the data. Consider this: aAC (used in MP4, YouTube, Apple Music) does better at the same bitrate. Opus (used in Discord, WhatsApp, WebRTC) is leading for speech and music at low bitrates Simple as that..

Lossless codecs (FLAC, ALAC, APE) compress ~40–60% with zero quality loss. Here's the thing — use them for archiving. Use Opus or AAC for distribution Nothing fancy..

Data encoding: the one you see in URLs and APIs

Base64. Percent

Data encoding: the one you see in URLs and APIs

Base64 is perhaps the most ubiquitous “encoding” you’ll encounter outside of human‑readable text. It takes binary data—any sequence of bytes—and maps it to a 64‑character alphabet (A–Z a–z 0–9 + /) plus a padding character (=). The process is simple: every three input bytes (24 bits) are split into four 6‑bit groups, each of which indexes into the alphabet. The output is roughly 33 % larger than the original, but it guarantees that every byte can be safely transmitted through a channel that expects only printable ASCII.

Why does Base64 show up everywhere? In practice, binary blobs (images, cryptographic keys, serialized objects) can’t be dropped into those streams without risking corruption. Think about it: because many transport layers—email (SMTP), HTTP multipart bodies, data URIs, JSON payloads, even QR codes—were designed around textual conventions. Base64 provides a loss‑less, reversible way to embed that binary payload as plain text.

Percent‑encoding (often called URL encoding) is the close cousin you meet when dealing with query strings, path segments, or form data. It works by representing any byte whose value falls outside the “safe” ASCII set as a % followed by two hexadecimal digits. Here's one way to look at it: a space becomes %20, a question mark becomes %3F. Unlike Base64, percent‑encoding preserves the original byte boundaries, which makes it ideal for constructing URLs that remain syntactically valid while still conveying arbitrary data.

Both Base64 and percent‑encoding are encodings in the strict sense: they are reversible transformations that map one set of values onto another. They are not compression methods—they increase size intentionally to achieve compatibility. That said, they are essential tools in a programmer’s toolbox, especially when interacting with web APIs, storing binary blobs in databases, or embedding data in JSON Web Tokens (JWTs).

People argue about this. Here's where I land on it.

Other notable data encodings

• Hexadecimal (0–9 a–f) is a compact, human‑readable representation of binary data, often used for debugging or in low‑level protocols. It is a subset of Base64 when the latter’s alphabet is restricted to 0–9 A–F.
• Quoted‑Printable is used in SMTP for encoding data that contains non‑ASCII characters but needs to stay mostly human‑readable. It replaces each non‑ASCII byte with =XX (hex) and encodes line breaks with =\r\n.
• UUencoding and BinHex were popular in early Unix and Mac OS environments for attaching binary files to email. They are largely obsolete now but illustrate the same principle: transform binary into printable characters.

All of these encodings share a common goal: bridge the gap between binary reality and textual constraints. Understanding their mechanics helps you choose the right tool for the job—whether you need a space‑efficient representation (Base64) or a URL‑safe string (percent‑encoding) Small thing, real impact. And it works..

Conclusion

Encoding is the invisible scaffolding that lets divergent systems speak the same language. Which means from the character encodings that turn raw bytes into readable text—ASCII, UTF‑8, UTF‑16—through data encodings that safely shuttle binary payloads across text‑only channels (Base64, percent‑encoding), to media encodings that shrink massive video and audio streams into consumable formats (H. 264, H.265, AAC, Opus), each layer solves a specific compatibility problem.

When you design a new system, ask yourself:

1. What is the underlying data type?

   • Text? Choose UTF‑8.
   • Binary blob? Wrap it in Base64 or store it as a BLOB.
   • Media? Pick the right codec and container for your quality‑vs‑size trade‑off.

2. Where will the data travel?

   • Over HTTP? Use UTF‑8 for headers, percent‑encoding for query parameters, Base64 for body payloads.
   • In a file? Choose a container that matches the codec you intend to use.

3. What constraints exist?

   • Legacy systems may force UTF‑16 or a specific container; isolate the conversion at the boundary.
   • Bandwidth or storage is limited; take advantage of modern lossy compression (AV1, Opus) to get the most fidelity per bit.

By treating encoding as a first‑class design decision—not an afterthought—you avoid the pitfalls that have plagued countless projects: corrupted data, ambiguous specifications, and interoperability nightmares. The right encoding strategy makes your data portable, future‑proof, and efficiently representable, no matter how disparate the systems that encounter it Small thing, real impact..