CODEWITHSUNDEEP
perlencodingendiannessutf-16
Hans Ginzel
Hans Ginzel

Reputation: 8830

Perl: Open output file in same endianess as input file – UTF-16be vs. UTF-16le

When Perl opens an UTF-16 encoded file,

open my $in, "< :encoding(UTF-16)", "text-utf16le.txt" or die "Error $!\n";

it automatically detects the endianess thanks Byte Order Mark.

But when I open file for writing

open my $out, "> :encoding(UTF-16)", "output.txt" or die "Error $!\n";

Perl opens it as big endian by default.

How to specify to open output file in the same endianness as input file, please?

How to get endianness/encoding from the input file handle $in? PerlIO::get_layers($in) returns among other layers encoding(UTF-16).

Upvotes: 1

Views: 137

Answers (1)

ikegami
ikegami

Reputation: 385764

You'll have to read the BOM yourself.

use IO::Unread qw( unread );

open(my $fh_in, "<:raw", $qfn)
   or die;

my $rv = read($fh_in, my $buf, 4);
defined($rv)
   or die;

my $encoding;
my $bom_present;
if    ($buf =~ s/^\x00\x00\xFE\xFF//) { $encoding = 'UTF-32be'; $bom_present = 1; }
elsif ($buf =~ s/^\xFF\xFE\x00\x00//) { $encoding = 'UTF-32le'; $bom_present = 1; }
elsif ($buf =~ s/^\xFE\xFF//        ) { $encoding = 'UTF-16be'; $bom_present = 1; }
elsif ($buf =~ s/^\xFF\xFE//        ) { $encoding = 'UTF-16le'; $bom_present = 1; }
elsif ($buf =~ s/^\xEF\xBB\xBF//    ) { $encoding = 'UTF-8';    $bom_present = 1; }
else {
   $encoding = 'UTF-8';
   $bom_present = 0;
}

unread($fh_in, $buf) if length($buf);

binmode($fh_in, ":encoding($encoding)");
binmode($fh_in, ":crlf") if $^O eq 'MSWin32';

But someone's already done that for you:

use File::BOM qw( open_bom );

my $encoding = open_bom(my $fh_in, $qfn, ':encoding(UTF-8)');

Upvotes: 5

Related Questions