CODEWITHSUNDEEP
perlutf-8
Brian Murphy
Brian Murphy

Reputation: 11

UTF-8 Coding Failure

Per a recommendation on this post:

I used:

use utf8;
use open ':encoding(utf8)';
binmode(STDOUT, ":utf8");
use open IN => ":encoding(utf8)", OUT => ':utf8';
use Encode;

It works when I do a search on my French

http://french.godsplanforlife.org/cgi_use/search.html page but fails on my Romanian page. http://romanian.godsplanforlife.org/cgi_use/search.html The special Romanian characters get switched from correct to incorrect when I do a search.

Here is the Perl code for search.pl It does the search and the printing of the search results at the bottom of the search page:

#!/usr/bin/perl
#search.pl
use utf8;
use open ':encoding(utf8)';

binmode(STDOUT, ":utf8");
use open IN => ":encoding(utf8)", OUT => ':utf8';

use Encode;

# The next three lines import special modules.
use CGI;
use CGI::Carp qw(fatalsToBrowser);
use File::Find;

$cgi=new CGI();

print $cgi->header();

$search_term = $cgi->param('search_term');
$page        = $cgi->param('page');
#Make the search term utf8 encoded.  
$search_term = decode_utf8( $search_term );

#The root directory is defined by the web hosting company.
# In this case it is Bluehost using Linux servers.
$root_dir = "/home2/godspla1/public_html/romanian";

$root_dir =~ s|/$||; #get rid of trailing slash

$html_lines= "";

#Specify directories to avoid searching.
$excluded = "cgi-bin|cgi_use|derived|images|_notes|_overlay|vti|_vti_cnf";

#Walk the directory tree;
#open the file and look for the term.
#See http://perldoc.perl.org/File/Find.html for the "find" function.
#\&search refers to the subroutine search() that will do the searching.
find( \&search, $root_dir ) if $search_term;

$html_lines ||= "<tr><td>No results found</td></tr>";

$search_results = qq{<table border="0" width="100%" align="center">}
                   .$html_lines.qq{</table>}; 

#Open the requested page to put in the results.
open (RESULTS, "$root_dir/$page") 
or die "Can't open results page ($root_dir/$page): $!";

#Substitute the search results and replace the search term too.
# see http://www.gossland.com/perlcourse/intro/flow for while loops.
while ( <RESULTS> ) {

#Move the point of printing insertion down to the results area.     
    s{<!-- search_results -->}{$search_results};

    s{name="search_term"\s*?value=""}
     {name="search_term" value="$search_term"};
     print;
}
close RESULTS;

#--This subroutine uses the find command on line 28 to find the search term.
sub search() {

    $seen = 0;

    $URL = $File::Find::name;
# !~ means not equal
# -f means the file is a normal file
#Exclude the exluded directories from the search. Files must be html.
    if ( $URL !~ m/$excluded|sidebar|footer|vti/ and -f and /.html?/ ) {

        $file = $_;
        open FILE, $file;
        @lines = <FILE>;
        close FILE;

#Grab the title, and the file name.  
#Each element ($_) of the @lines array is one paragraph from file.
        for ( @lines ) {

            $title = $1 if m|<title>(.*?)</title>|;
#The Q and the E are delimiters to escape interpretation.
#Increment $seen by one, which makes it true, if the match is seen.
            $seen++ if /\Q$search_term\E/i;
            $seen-- if m/\Q$search_term<\/a>\E/i;
        }

        if ( $seen ) {
            $URL =~ s|$root_dir||; 

#Format the found results into URL, title.
            $html_lines .= qq{<tr><td><a href="$URL">$URL</a>};
            $html_lines .= qq{</td><td>$title</td></tr>\n};
        }
    }
}

Upvotes: 1

Views: 146

Answers (1)

H&#229;kon H&#230;gland
H&#229;kon H&#230;gland

Reputation: 40718

To correctly read UTF8 data from the HTTP_POST from the browser, you can use either use CGI; and decode later: 

use CGI;
binmode STDIN;
use Encode;
$search_term = $cgi->param('search_term');
$search_term = decode_utf8( $search_term );

or use CGI qw ( -utf8 ); :

use CGI qw ( -utf8 );
binmode STDIN;
$search_term = $cgi->param('search_term');

To correctly read, modify and print (to STDOUT) the UTF8-encoded template file used by the CGI script to generate output, you should enable UTF8 enocding on file read, and on output to STDOUT:

use open IN => ":encoding(utf8)";
binmode STDOUT, ":utf8";

Finally, you need to tell the browser that the received data contains UTF8:

$cgi->header(-type => 'text/html', -charset => 'utf-8');

From your script, it seems that the problem is mainly related to the last point.. (You are missing -charset => 'utf-8' )

Upvotes: 1

Related Questions