Convert from ANSI to UTF-8 Encoding

Question

I have an ASP.Net 4.0 web application, written in C#, and one of the things it does is write a file to the file system based on data pulled from a SQL Server 2012 query. Sometimes the user input contains characters from French, frequently cut and pasted from Microsoft Word and therefore in ANSI encoding. This file my web application creates is then loaded into another program via code outside of my control, i.e., not in my web application. The problem is that this second program requires UTF-8 encoding. I've written code to convert my program's output to UTF-8, but it's still not loading correctly, so I think I'm doing something wrong. Here's my code:

protected void writeToClientFile(DataSet ClientGenl, DataSet ClientBus, DataSet ClientBill)
    {
        FileStream fileStream = null;
        string fileName = "ClientTest.txt";
        string pathName = ConfigurationSettings.AppSettings["EliteFilePath"].ToString();
        try
        {
            using (new KLClassLibrary.Impersonator(proxyaccount, domain, password))
            {
                fileStream = OpenASAP(pathName + fileName, 10);
                using (TextWriter tw = new StreamWriter(fileStream))
                {
                    foreach (DataRow rowGeneral in ClientGenl.Tables[0].Rows)
                    {
                        string fileTextGeneral = "CLNUM:" + rowGeneral["clnum"].ToString().toEliteInput();
                        byte[] originalBytes = Encoding.Default.GetBytes(fileTextGeneral);
                        byte[] convertedBytes = Encoding.Convert(Encoding.Default, Encoding.UTF8, originalBytes);
                        char[] convertedChars = new char[Encoding.UTF8.GetCharCount(convertedBytes, 0, convertedBytes.Length)];
                        Encoding.UTF8.GetChars(convertedBytes, 0, convertedBytes.Length, convertedChars, 0);
                        string convertedString = new string(convertedChars);
                        tw.WriteLine(convertedString);
                    }
                }
            }
        }
        catch (Exception ex)
        {
            throw ex;
        }
        finally
        {
            if (fileStream != null)
                fileStream.Dispose();
        }
    }
FileStream OpenASAP(string path, int maxTries)
    {
        FileStream fs = null;
        bool lastResult = false;
        int count = 0;
        while ((lastResult == false) && count < maxTries)
        {
            lastResult = TryOpen(path, out fs);
            Thread.Sleep(100);
            count++;
        }
        if (!lastResult || count >= maxTries)
        {
            throw new Exception("The file is being written to");
        }
        return fs;
    }
bool TryOpen(string FileWithPath, out FileStream fs)
    {
        try
        {
            fs = File.Open(FileWithPath, FileMode.Append, FileAccess.Write, FileShare.None);
            return true;
        }
        catch (Exception ex)
        {
            fs = null;
            return false;
        }
    }

Answer 1

You could derive the original encoding from the StreamReader.CurrentEncoding, and then convert the Encoding using Encoding.Convert(), specifying the derived encoding as the source encoding and Encoding.UTF8 as the destination encoding.

This way, you let the StreamReader decide which encoding fits the content of the source file.
It will probably be Unicode (UTF16 in Windows), converted from the Default encoding.

string TextDestinaton = string.Empty;

using (FileStream FileOrigin = new FileStream(@"[SomeSourceFile]", 
                                   FileMode.Open, FileAccess.Read, FileShare.None))
using (StreamReader orgReader = new StreamReader(FileOrigin))
{
    Encoding OriginalEncoding = Encoding.GetEncoding(orgReader.CurrentEncoding.CodePage);
    byte[] OriginalBytes = OriginalEncoding.GetBytes(orgReader.ReadToEnd());
    byte[] DestinationBytes = Encoding.Convert(OriginalEncoding, Encoding.UTF8, OriginalBytes, 0, OriginalBytes.Length);

    using (MemoryStream memstream = new MemoryStream(DestinationBytes, 0, DestinationBytes.Length))
    using (StreamReader destReader = new StreamReader(memstream, Encoding.UTF8))
    {
        memstream.Position = 0;
        TextDestinaton = destReader.ReadToEnd();
    };
}

As a note: if the original text contained characters that did not fit the Local Encoding (when the text was acquired), and no specific Encoding was used to preserve the source CodePage mapping, the source text could be compromised.