CODEWITHSUNDEEP
phpdatabaseutf-8character-encoding
ThisLanham
ThisLanham

Reputation: 745

Arabic Character Encoding Issue: UTF-8 versus Windows-1256

Quick Background: I inherited a large SQL dump file containing a combination of English and Arabic text, and (I think) it was originally exported using 'latin1'. I changed all occurrences of 'latin1' to 'utf8' prior to importing the file. The Arabic text didn't appear correctly in phpMyAdmin (which I guess is normal), but when I loaded the text to a web page with the following, everything looked good and the arabic text displayed perfectly.:

<meta http-equiv='Content-Type' content='text/html; charset=windows-1256'/>

Problem: My client is picky and doesn't want to change his

<meta http-equiv="Content-Type" content="text/html; charset=UTF-8"/>

to the 'Windows-1256' equivalent. I didn't think this would be a problem, but when I changed the charset value to 'UTF-8', all the Arabic characters appeared as diamonds with question marks. Shouldn't UTF-8 display Arabic text correctly?

Here are a few notes about my database configuration:

Upvotes: 8

Views: 45686

Answers (5)

mostafa khansa
mostafa khansa

Reputation: 302

In order to display Arabic characters correctly, you need to convert your PHP file to utf-8 without Bom. This happened to me, Arabic characters were displayed as diamonds, but conversion to utf-8 without bom will solve this problem.

Upvotes: 0

Khalidco
Khalidco

Reputation: 13

I seems that the db is configured as UTF8, but the data itself is extended ascii. If the data is converted to UTF8, it will display correctly in content type set to UTF8

Upvotes: 0

Michael Dillon
Michael Dillon

Reputation: 32392

I think you need to go back to square one. It sounds like you have a database dump in Win-1256 encoding and you want to work with it in UTF-8 from now on. It also sounds like you are using PHP but you have lots of irrelevant tags on your question and are missing the most important one, PHP.

First, you need to convert the text dump into UTF-8 and you should be able to do that with PHP. Chances are that your conversion script will have two steps, first read the Win-1256 bytes and decode them into internal Unicode text strings, then encode the Unicode text strings into UTF-8 bytes for output to a new text file.

Once you have done that, redo the database import as you did before, but now you have correctly encoded the input data as UTF-8.

After that it should be as simple as reading the database and rendering a web page with the correct UTF-8 encoding.

P.S. It is actually possible to reencode the data every time you display it, but that does not solve the problem of having a database full of incorrectly encoded data.

Upvotes: 2

ikegami
ikegami

Reputation: 386706

We can't find the error in your code if you don't show us your code, so we're very limited in how we can help you.

You told the browser to interpret the document as being UTF-8 rather than Windows-1256, but did you actually change the encoding used from Windows-1256 to UTF-8?

For example,

$ cat a.pl
use strict;
use warnings;
use feature qw( say );
use charnames ':full';

my $enc = $ARGV[0] or die;
binmode STDOUT, ":encoding($enc)";

print <<"__EOI__";
<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=$enc">
<title>Foo!</title>
</head>
<body dir="rtl">
\N{ARABIC LETTER ALEF}\N{ARABIC LETTER LAM}\N{ARABIC LETTER AIN}\N{ARABIC LETTER REH}\N{ARABIC LETTER BEH}\N{ARABIC LETTER YEH}\N{ARABIC LETTER TEH MARBUTA}
</body>
</html>
__EOI__

$ perl a.pl UTF-8 > utf8.html

$ perl a.pl Windows-1256 > cp1256.html

Upvotes: 2

Jukka K. Korpela
Jukka K. Korpela

Reputation: 201896

If the document looks right when declared as windows-1256 encoded, then it most probably is windows-1256 encoded. So it was apparently not exported using latin1—which would have been impossible, since latin1 has no Arabic letters.

If this is just about a single file, then the simplest way is to convert it from windows-1256 encoding to utf-8 encoding, using e.g. Notepad++. (Open the file in it, change the encoding, via File format menu, to Arabic, windows-1256. Then select Convert to UTF-8 in the File format menu and do File → Save.)

Windows-1256 and UTF-8 are completely different encodings, so data gets all messed up if you declare windows-1256 data as UTF-8 or vice versa. Only ASCII characters, such as English letters, have the same representation in both encodings.

Upvotes: 4

Related Questions