CODEWITHSUNDEEP
perltestingutf-8
melo
melo

Reputation: 315

Why do my Perl tests fail with use encoding 'utf8'?

I'm puzzled with this test script:

#!perl

use strict;
use warnings;
use encoding 'utf8';
use Test::More 'no_plan';

ok('áá' =~ m/á/, 'ok direct match');

my $re = qr{á};
ok('áá' =~ m/$re/, 'ok qr-based match');

like('áá', $re, 'like qr-based match');

The three tests fail, but I was expecting that the use encoding 'utf8' would upgrade both the literal áá and the qr-based regexps to utf8 strings, and thus passing the tests.

If I remove the use encoding line the tests pass as expected, but I can't figure it out why would they fail in utf8 mode.

I'm using perl 5.8.8 on Mac OS X (system version).

Upvotes: 9

Views: 3026

Answers (3)

Michael
Michael

Reputation: 8526

The Test::More documentation contains a fix for this issue, which I just found today (and this entry shows higher in the googles).

utf8 / "Wide character in print"

If you use utf8 or other non-ASCII characters with Test::More you might get a "Wide character in print" warning. Using binmode STDOUT, ":utf8" will not fix it. Test::Builder (which powers Test::More) duplicates STDOUT and STDERR. So any changes to them, including changing their output disciplines, will not be seem by Test::More. The work around is to change the filehandles used by Test::Builder directly.

my $builder = Test::More->builder;
binmode $builder->output,         ":utf8";
binmode $builder->failure_output, ":utf8";
binmode $builder->todo_output,    ":utf8";

I added this bit of boilerplate to my testing code and it works a charm.

Upvotes: 2

Aristotle Pagaltzis
Aristotle Pagaltzis

Reputation: 117959

Do not use the encoding pragma. It’s broken. (Juerd Waalboer gave a great talk where he mentioned this at YAPC::EU 2k8.)

It does at least two things at once that do not belong together:

  1. It specifies an encoding for your source file.
  2. It specifies an encoding for your file input/output.

And to add injury to insult it also does #1 in a broken fashion: it reinterprets \xNN sequences as being undecoded octets as opposed to treating them like codepoints, and decodes them, preventing you from being able to express characters outside the encoding you specified and making your source code mean different things depending on the encoding. That’s just astonishingly wrong.

Write your source code in ASCII or UTF-8 only. In the latter case, the utf8 pragma is the correct thing to use. If you don’t want to use UTF-8, but you do want to include non-ASCII charcters, escape or decode them explicitly.

And use I/O layers explicitly or set them using the open pragma to have I/O automatically transcoded properly.

Upvotes: 18

Leon Timmermans
Leon Timmermans

Reputation: 30225

It works fine on my computer (on perl 5.10). Maybe you should try replacing that use encoding 'utf8' with use utf8.

What version of perl are you using? I think older versions had bugs with UTF-8 in regexps.

Upvotes: 2

Related Questions