Skip to Content
Course content

187: utf8 Package for Rune Handling

Click on the "Edit" button in the top corner of the screen to edit your slide content.

I've seen this mistake more times than I can count, even from developers who have been using Go for a while: assuming that len(str) tells you how many characters are in a string. It feels intuitive, right? If you have a word, you want to know how long it is. But in Go, strings are read-only slices of bytes, not slices of characters.

The "len() equals character count" trap

Let's look at a concrete example. Suppose you're building a username validator and you want to make sure a name isn't too long. You might write something like this:

package main

import "fmt"

func main() {
    username := "Gopher ʕ•ᴥ•ʔ"
    fmt.Println("Length:", len(username)) 
}

If you run this, you'll expect to see 13 (the number of visual characters). Instead, you'll get 21. Why? Because the "bear" emojis and special characters in that string are encoded using multiple bytes in UTF-8. len() is simply counting those bytes. If you use this for a database column limit or a UI constraint, you're going to end up cutting users off mid-character, which results in those ugly replacement characters (like ) appearing in your app.

Counting actual runes with the utf8 package

To get the actual number of Unicode code points—which we call runes in Go—you need the unicode/utf8 package. I always reach for utf8.RuneCountInString() when the visual length is what actually matters for the business logic.

package main

import (
    "fmt"
    "unicode/utf8"
)

func main() {
    username := "Gopher ʕ•ᴥ•ʔ"
    
    byteLen := len(username)
    runeLen := utf8.RuneCountInString(username)
    
    fmt.Printf("Bytes: %d\n", byteLen) // 21
    fmt.Printf("Runes: %d\n", runeLen) // 13
}

Notice that RuneCountInString has to iterate through the string to decode the UTF-8 sequences, making it an O(n) operation, whereas len() is O(1) because it just reads the slice header. I usually tell my juniors: use len() for memory allocation and buffer sizing, but use utf8 for anything the user actually sees.

Ensuring your bytes actually make sense

Beyond just counting, there's a risk when you're receiving data from an external API or a raw TCP socket. Go doesn't force strings to be valid UTF-8; a string is just a sequence of bytes. You can technically cast a slice of random garbage bytes into a string, and Go won't complain—until you try to print it or process it.

This is where utf8.ValidString() becomes your best friend. I use this as a guard clause at the edge of my system to reject malformed data before it hits my internal logic.

package main

import (
    "fmt"
    "unicode/utf8"
)

func main() {
    // A slice of bytes that is NOT valid UTF-8
    badBytes := []byte{0xff, 0xfe, 0xfd}
    badString := string(badBytes)

    if !utf8.ValidString(badString) {
        fmt.Println("Warning: This string contains invalid UTF-8 sequences!")
    }
}

If you're doing heavy-duty manipulation—like manually decoding a stream of bytes one character at a time—you can also use utf8.DecodeRuneInString(). It returns the first rune and its size in bytes, allowing you to "step" through a string without converting the whole thing into a []rune slice (which would allocate a whole new piece of memory).




📋 Practical Task

Build a Unicode-Aware Character Limiter

You are tasked with writing a function for a social media profile bio. The requirement is that the bio must be between 10 and 100 characters (runes), regardless of how many bytes those characters take up. Additionally, the function must reject the input entirely if it contains invalid UTF-8 sequences.

Implement a function with the following signature:

func ValidateBio(bio string) (bool, error)

Requirements:

  • Use utf8.ValidString to ensure the input is valid UTF-8. If not, return false and an error.
  • Use utf8.RuneCountInString to check the length.
  • Return true, nil if the length is between 10 and 100 inclusive.
  • Return false, nil if the length is outside that range.

Test your function with these cases:

  • "Hello World!" (Valid, within range)
  • "Hi" (Valid, too short)
  • "Gopher ʕ•ᴥ•ʔ is a very happy little creature that loves Go!" (Valid, within range)
  • A string created from invalid bytes: string([]byte{0xff, 0xfe, 0xfd}) (Invalid UTF-8)
Rating
0 0

There are no comments for now.

to be the first to leave a comment.