CODEWITHSUNDEEP
regexperlcharacterdata-conversioncjk
robasia
robasia

Reputation: 113

Converting full-width characters to half-width characters

I have a program for converting full width characters to half width. It works fine, except for the number zero. Full-width zero is not converting to half-width zero.

Perl

use strict;
use warnings;
use warnings qw(FATAL utf8);
use utf8;
use feature qw(unicode_strings);
use open qw(:std :utf8);

unless ( @ARGV == 2 ) { 
    print "Usage: script.pl input_file output_file\n";
    exit;
}

my %fwhw = (
        '０' => '0', '１' => '1', '２' => '2', '３' => '3', '４' => '4',
        '５' => '5', '６' => '6', '７' => '7', '８' => '8', '９' => '9',
        'Ａ' => 'A', 'Ｂ' => 'B', 'Ｃ' => 'C', 'Ｄ' => 'D', 'Ｅ' => 'E',
        'Ｆ' => 'F', 'Ｇ' => 'G', 'Ｈ' => 'H', 'Ｉ' => 'I', 'Ｊ' => 'J',
        'Ｋ' => 'K', 'Ｌ' => 'L', 'Ｍ' => 'M', 'Ｎ' => 'N', 'Ｏ' => 'O',
        'Ｐ' => 'P', 'Ｑ' => 'Q', 'Ｒ' => 'R', 'Ｓ' => 'S', 'Ｔ' => 'T',
        'Ｕ' => 'U', 'Ｖ' => 'V', 'Ｗ' => 'W', 'Ｘ' => 'X', 'Ｙ' => 'Y',
        'Ｚ' => 'Z', 'ａ' => 'a', 'ｂ' => 'b', 'ｃ' => 'c', 'ｄ' => 'd',
        'ｅ' => 'e', 'ｆ' => 'f', 'ｇ' => 'g', 'ｈ' => 'h', 'ｉ' => 'i',
        'ｊ' => 'j', 'ｋ' => 'k', 'ｌ' => 'l', 'ｍ' => 'm', 'ｎ' => 'n',
        'ｏ' => 'o', 'ｐ' => 'p', 'ｑ' => 'q', 'ｒ' => 'r', 'ｓ' => 's',
        'ｔ' => 't', 'ｕ' => 'u', 'ｖ' => 'v', 'ｗ' => 'w', 'ｘ' => 'x',
        'ｙ' => 'y', 'ｚ' => 'z', '－' => '-', '、' => ', ', '　' => ' ',
         '／' => '/',);

sub slurp {
    my $file = shift;
    open my $fh_read, '<', $file or die "Could not open file: $!";
    return do {local $/; <$fh_read>};
}

sub convert {
    my $sub_string = shift;
    $sub_string =~ s/(.)/$fwhw{$1}?$fwhw{$1}:$1/seg;
    return $sub_string;
}

my $string = slurp($ARGV[0]);

$string =~ s/<target>\s*<g id="\d+">\K(.*?)(?=<\/g>\s*<\/target>)/convert($1)/seg;

open my $fh_write, ">", $ARGV[1] or die "Could not open file: $!";

print $fh_write $string;

close $fh_write;

Here is what I have tried

What I haven't tried yet:

If anyone has any suggestions or sees my mistake, I would be very grateful for the guidance.

Upvotes: 2

Views: 3331

Answers (2)

phuclv
phuclv

Reputation: 41962

You don't need such a huge dictionary with lots of supporting functions like that. Just a simple sed is enough

halfwidth='!"#$%&'\''()*+,-.\/0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~｟｠¢£¬¯¦¥₩ '
fullwidth='！＂＃＄％＆＇（）＊＋，－．／０１２３４５６７８９：；＜＝＞？＠ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ［＼］＾＿｀ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ｛｜｝～｟｠￠￡￢￣￤￥￦　'
sed -ie "y/$fullwidth/$halfwidth/" your_file

If you want to do that in perl it's pretty simple too

perl -Mutf8 -i -C -pe 'BEGIN{ use open qw/:std :utf8/; } tr#！＂＃＄％＆＇（）＊＋，－．／０１２３４５６７８９：；＜＝＞？＠ＡＢＣＤＥＦＧＨＩＪＫＬＭＮＯＰＱＲＳＴＵＶＷＸＹＺ［＼］＾＿｀ａｂｃｄｅｆｇｈｉｊｋｌｍｎｏｐｑｒｓｔｕｖｗｘｙｚ｛｜｝～｟｠￠￡￢￣￤￥￦　#!"\#$%&'\''()*+,-.\/0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\\]^_`abcdefghijklmnopqrstuvwxyz{|}~｟｠¢£¬¯¦¥₩ # your_file'

Upvotes: 1

ikegami
ikegami

Reputation: 386541

Since $fwhw{'０'} returns 0, and since 0 is false, the replacement doesn't occur. Replace

$sub_string =~ s/(.)/$fwhw{$1}?$fwhw{$1}:$1/seg;

with

$sub_string =~ s/(.)/exists($fwhw{$1})?$fwhw{$1}:$1/seg;

If that still doesn't work, use sprintf "%vX", $str to see what you really have.

By the way,

sub convert {
    my $sub_string = shift;
    $sub_string =~ s/(.)/exists($fwhw{$1})?$fwhw{$1}:$1/seg;
    return $sub_string;
}

would be much faster if replaced with

sub convert {
    state $chars = join '', keys(%fwhw);
    state $re = qr/([\Q$chars\E])/;
    return $_[0] =~ s/$re/$fwhw{$1}/gr;
}

Faster yet,

sub convert {
    state $s = join '', keys(%fwhw);
    state $r = join '', values(%fwhw);
    state $tr = eval("sub { $_[0] =~ tr/\Q$s\E/\Q$r\E/r }");
    return $tr->($_[0]);
}

Upvotes: 4

Related Questions