CODEWITHSUNDEEP
perlencoding
Yuhao
Yuhao

Reputation: 1610

Perl non-English Character

See this piece of perl code:

#!/usr/bin/perl -w -CS

use feature 'unicode_strings';

open IN, "<", "wiki.txt";
open OUT, ">", "wikicorpus.txt";

binmode( IN,  ':utf8' );
binmode( OUT, ':utf8' );

## Condition plain text English sentences or word lists into a form suitable for constructing a vocabulary and language model

while (<IN>) {

  # Remove starting and trailing tags (e.g. <s>)
  # s/\<[a-z\/]+\>//g;

  # Remove ellipses 
  s/\.\.\./ /g;

  # Remove unicode 2500 (hex E2 94 80) used as something like an m-dash between words
  # Unicode 2026 (horizontal ellipsis)
  # Unicode 2013 and 2014 (m- and n-dash)
  s/[\x{2500}\x{2026}\x{2013}\x{2014}]/ /g;

  # Remove dashes surrounded by spaces (e.g. phrase - phrase)
  s/\s-+\s/ /g;

  # Remove dashes between words with no spaces (e.g. word--word)
  s/([A-Za-z0-9])\-\-([A-Za-z0-9])/$1 $2/g;

  # Remove dash at a word end (e.g. three- to five-year)
  s/(\w)-\s/$1 /g;

  # Remove some punctuation
  s/([\"\?,;:%???!()\[\]{}<>_\.])/ /g;

  # Remove quotes
  s/[\p{Initial_Punctuation}\p{Final_Punctuation}]/ /g;

  # Remove trailing space
  s/ $//;

  # Remove double single-quotes 
  s/'' / /g;
  s/ ''/ /g;

  # Replace accented e with normal e for consistency with the CMU pronunciation dictionary
  s/?/e/g;

  # Remove single quotes used as quotation marks (e.g. some 'phrase in quotes')
  s/\s'([\w\s]+[\w])'\s/ $1 /g;

  # Remove double spaces
  s/\s+/ /g;

  # Remove leading space
  s/^\s+//;

  chomp($_);

  print OUT uc($_) . "\n";
#  print uc($_) . " ";
} print OUT "\n";

It seems that there is a non-english character on line 49, namely the line s/?/e/g;. So when I run this, warning come out that Quantifier follows nothing in regex;.

How can I deal with this problem? How to make perl recognize the character? I have to run this code with perl 5.10.

Another little question is that what is the meaning of the "-CS" in the 1st line.

Thanks to all.

Upvotes: 0

Views: 916

Answers (2)

tripleee
tripleee

Reputation: 189317

As per the comment line just before the erroneous line, the character to be replaced is an accented "e"; presumably what is meant is e with an acute accent: "é". Assuming your input is Unicode, it can be represented in Perl as \x{00E9}. See also http://www.fileformat.info/info/unicode/char/e9/index.htm

I guess you copy/pasted this script from a web page on a server which was not properly configured to display the required character encoding. See further also http://en.wikipedia.org/wiki/Mojibake

Upvotes: 1

ddyer
ddyer

Reputation: 1788

I think your problem is that your editor doesn't handle unicode characters, so the program is trashed before it even gets to perl, and as this apparently isn't your program, it was probably trashed before it got to you.

Until the entire tool chain handles unicode correctly, you have to be careful to encode non-ascii characters in a way that preserves them. It's a pain, and no simple solutions exist. Consult your perl manual for how to embed unicode characters safely.

Upvotes: 1

Related Questions