CODEWITHSUNDEEP
c#.netutf-8
Alireza Noori
Alireza Noori

Reputation: 15243

Download a webpage in UTF-8

I'm using the code below to download this XML file:

private async static Task<string> DownloadPageAsync(string url)
{
    try
    {
        HttpClientHandler handler = new HttpClientHandler();
        handler.UseDefaultCredentials = true;
        handler.AllowAutoRedirect = true;
        handler.UseCookies = true;
        HttpClient client = new HttpClient(handler);
        client.MaxResponseContentBufferSize = 10000000;
        HttpResponseMessage response = await client.GetAsync(url);
        response.EnsureSuccessStatusCode();

        string responseBody = response.Content.ReadAsString();
        return responseBody;
    }
    catch (Exception ex)
    {
        return "error" + ex.Message;
    }
}

but the document I'm getting seems to have encoding problems. Although the document is not well formatted, I'm guessing my downloaded webpage is not in UTF-8 either. How can I return a UTF-8 string? Thanks.

Upvotes: 0

Views: 664

Answers (2)

BLUEPIXY
BLUEPIXY

Reputation: 40145

your link encoding is iso-8859-1.

use

XmlDocument.Load(uriString)

or

XDocument.Load(uriString)

Upvotes: 1

Oded
Oded

Reputation: 498914

I suggest using the HTML Agility Pack to download and parse the document for you - it will automatically detect the encoding (where possible), so this shouldn't be a problem for you.

If this is not an option, you need to know what encoding the document is using then transform it to UTF8 using the Encoding classes to convert from the original encoding to UTF8.

Upvotes: 1

Related Questions