Double Encoding, Unicode, and Other URL Encoding Gotchas

URL encoding seems straightforward until you encounter nested encoding, Unicode normalization, or OAuth callback URLs. These edge cases cause silent failures that are notoriously hard to debug because the encoded values look correct at a glance.

Solving the Tricky Cases

When a URL-encoded value is placed inside another URL, you may need to encode it again — this is "double encoding" and it's legitimate. For Unicode, ensure your encoding function handles UTF-8 correctly — older functions may produce Latin-1 output. For OAuth callbacks, URL-encode the entire redirect_uri parameter value so its own ? and & characters don't break the outer URL. When debugging, decode the URL step by step to find where the encoding went wrong.

Why This Matters in 2026

Encoding is the invisible backbone of data interchange. Every time data moves between systems — URLs, HTML pages, API payloads, email attachments — encoding ensures it arrives intact. In 2026, with the TC39 Base64 proposal reaching stage 4 and Unicode handling remaining a persistent source of bugs, understanding encoding is more important than ever. Developers who grasp the differences between Base64, Base64URL, URL encoding, and HTML encoding avoid an entire class of production bugs.

Key Takeaways

• Base64 is for representing binary data as text — it is not encryption
• Use Base64URL instead of Base64 when the output goes in URLs or JWTs
• Use encodeURIComponent for query parameter values, encodeURI for full URLs
• HTML-encode user input before inserting it into HTML to prevent XSS
• The new Uint8Array.toBase64() replaces btoa() for proper binary support
• Always handle UTF-8 encoding correctly when working with non-ASCII text

Common Mistakes to Avoid

• Using btoa() with Unicode strings — it silently corrupts non-ASCII characters
• Putting standard Base64 (with +, /, =) directly in URLs without converting to Base64URL
• Using encodeURI instead of encodeURIComponent for query parameter values
• Treating Base64 as a security measure — it is trivially decodable by anyone
• Double-encoding values, producing %2520 instead of %20
• Not HTML-encoding user input before rendering, leading to XSS vulnerabilities

Best Practices

• Use Uint8Array.toBase64() and Uint8Array.fromBase64() in modern browsers
• Use Buffer.from(input, "base64") in Node.js for correct binary handling
• Always convert Base64 to Base64URL for URLs, JWTs, and HTTP headers
• HTML-encode all user input before inserting into HTML content or attributes
• Use encodeURIComponent for individual query parameters, encodeURI for full URLs
• Test encoding with Unicode text (emoji, CJK characters) to catch edge cases

Quick Reference

Here are the modern encoding and decoding patterns for 2026, using the new TC39 APIs where available:

// Modern Base64 encoding (TC39 stage 4)
const bytes = new TextEncoder().encode("Hello, World!");
const base64 = bytes.toBase64(); // "SGVsbG8sIFdvcmxkIQ=="
const decoded = new TextDecoder().decode(Uint8Array.fromBase64(base64));

// URL encoding
const param = encodeURIComponent("name=John & Sons");
const fullUrl = encodeURI("https://example.com/path with spaces");

// HTML encoding
function htmlEncode(str) {
  return str.replace(/&/g, "&amp;").replace(/</g, "&lt;")
    .replace(/>/g, "&gt;").replace(/"/g, "&quot;").replace(/'/g, "&#x27;");
}

Real-World Example

A common scenario: you need to send a binary image as Base64 in a JSON API payload, then display it in the browser. Here is how to do it correctly with proper encoding at each step:

// Convert image file to Base64 for API transport
async function imageToBase64(file) {
  const bytes = new Uint8Array(await file.arrayBuffer());
  return bytes.toBase64(); // URL-safe, handles binary correctly
}

// Display Base64 image in the browser
function base64ToImage(base64, mimeType) {
  return `data:${mimeType};base64,${base64}`;
}

// Usage
const b64 = await imageToBase64(fileInput.files[0]);
imgElement.src = base64ToImage(b64, "image/png");

Tools and Resources

• DevKit Base64 Encoder/Decoder — browser-based with URL-safe variant support
• DevKit URL Encode/Decode — encodeURI vs encodeURIComponent with live preview
• DevKit HTML Encoder/Decoder — entity encoding for XSS prevention
• DevKit Hash Generator — cryptographic hashing for data integrity
• CyberChef — advanced encoding and decoding Swiss Army knife

"Encoding is not encryption. Encoding is for transport. Encryption is for secrecy. Never confuse the two."

Encoding is fundamental to data interchange, and using the wrong type causes silent bugs that are notoriously hard to trace. Use the modern TC39 Base64 APIs, choose the right URL encoding function, always HTML-encode user input, and remember: encoding is for transport, not security.