CODEWITHSUNDEEP
unicodeencodinggoutf-8character-encoding
RandomGuyqwert
RandomGuyqwert

Reputation: 435

Go and UTF-8 encoding - Is conversion automatic?

I am making http requests using Go.

request, err := http.NewRequest("GET", url, nil)

This request, if successful, returns a response.

response, err := client.Do(request)

After receiving a response, I want to save the content.

content, err := ioutil.ReadAll(response.Body)
ioutil.WriteFile(destination, content, 0644)

I looked at the Headers of the responses.

response.Header.Get("Content-Type")

I saw the majority are already UTF-8 encoded, which is good. But there are some that have different encodings. I know Go has built in unicode support. Does that mean that if I write, for example, the content of a big-5 encoded page, it will be automatically converted to utf-8? Or do I need to manually decode using the big-5 encoding and re-encode using utf-8?

Basically, I want to ensure that everything that gets written is utf-8 encoded. What is the best way to achieve this?

Thanks!

Upvotes: 1

Views: 2248

Answers (1)

Volker
Volker

Reputation: 42413

What ioutil.ReadAll reads will be written with ioutil.WriteFile without any conversions whatsoever.

If you want to force UTF-8 encoded you will have to do the de-/encoding yourself, e.g. with the help of golang.org/x/text/encoding{,/charmap} and/or the unicode/utf{8,16} packages.

Be prepared for all sorts of ugliness and a lot of pain.

Upvotes: 1

Related Questions