Need to batch convert a large quantity of text files from ANSI to Unicode

Question

I have a lot of ANSI text files that vary in size (from a few KB up to 1GB+) that I need to convert to Unicode.

At the moment, this has been done by loading the files into Notepad and then doing "Save As..." and selecting Unicode as the Encoding. Obviously this is very time consuming!

I'm looking for a way to convert all the files in one hit (in Windows). The files are in a directory structure so it would need to be able to traverse the full folder structure and convert all the files within it.

I've tried a few options but so far nothing has really ticked all the boxes:

• ansi2unicode command line utility. This has been the closest to what I'm after as it processes files recursively in a folder structure...but it keeps crashing whilst running before it's finished converting.
• CpConverter GUI utility. Works OK to a point but struggles with multiple files in a folder structure - only seems to be able to handle files in one folder
• There's a DOS command that works OK on smaller files but doesn't seem to be able to cope with large files.
• Tried GnuWin sed utility but it crashes every time I try and install it

So I'm still looking! If anyone has any recommendations I'd be really grateful

Thanks...

Answer 1

When I transfered my personal website to my Ipad every html files bore some weird characters. The ipad server has another filesystem. I had to convert hundreds of html and je files from ansi to utf16 LE. This PowerShell snipet saved me a lot of time while I don't know the first thing about Powershell. The lame Copy/Paste method sufficed to run the script. I added the commands remove and rename item to suit my need (It’s a method i used in Liberty Basic with rename and kill).

I ll keep it carefully. Thank you

Answer 2

OK, so in case anyone else is interested, I found a way to do this using PowerShell:

Get-ChildItem "c:\some path\" -Filter *.csv -recurse | 
    Foreach-Object {
    Write-Host (Get-Date).ToString() $_.FullName
    Get-Content $_.FullName | Set-Content -Encoding unicode ($_.FullName  + '_unicode.csv')
}

This recurses through the entire folder structure and converts all CSV files to Unicode; the converted files are written to the same locations as the originals but with "unicode" appended to the filename. You can change the value of the -Encoding parameter if you want to convert to something different (e.g. utf-8).

It also outputs a list of all the files converted along with a timestamp against each