CODEWITHSUNDEEP
postgresqlperldbimojolicious
Eugen Konkov
Eugen Konkov

Reputation: 25153

Input is not UTF-8 encoded

my database is utf8 aware:

                                   List of databases
   Name    |     Owner     | Encoding |  Collate   |   Ctype    |
-----------+---------------+----------+------------+------------+
 tucha     | tucha_cleaner | UTF8     | en_US.utf8 | en_US.utf8 |

When I connect to it I set client_encoding:

my $hm_schema = App::Schema->connect( $dsn, $user, $pass, {
        AutoCommit => 1,
        RaiseError => 1,
        client_encoding => 'UTF8',
    }
);

The returned value, as far as I can see, is UTF8:

DBG>$value
["Нд", "Пн", "Вт", "Ср", "Чт", "Пт", "Сб"]

DBG>use Devel::Peek

DBG>Devel::Peek::Dump $value
SV = PVMG(0xfe41c20) at 0xfe079b0
  REFCNT = 1
  FLAGS = (POK,IsCOW,pPOK,UTF8)
  IV = 0
  NV = 0
  PV = 0xfe27550 "[\"\320\235\320\264\", \"\320\237\320\275\", \"\320\222\321\202\", \"\320\241\321\200\", \"\320\247\321\202\", \"\320\237\321\202\", \"\320\241\320\261\"]"\0 [UTF8 "["\x{41d}\x{434}", "\x{41f}\x{43d}", "\x{412}\x{442}", "\x{421}\x{440}", "\x{427}\x{442}", "\x{41f}\x{442}", "\x{421}\x{431}"]"]
  CUR = 56
  LEN = 58
  COW_REFCNT = 4
undef

But when I try to decode that string by decode_json from Mojo::JSON I get the error:

DBG> decode_json $value
ERROR: Input is not UTF-8 encoded at ...

Why I get that error and how to fix it?

Upvotes: 2

Views: 1614

Answers (1)

ikegami
ikegami

Reputation: 385917

The first 5 characters of your string are the following (in hex):

5B 22 41D 434 22

Character encodings such as UTF-8 are means of representing code points using bytes, and two of those characters aren't bytes, so your string can't possibly be JSON encoded using UTF-8.

It appears that you have a decoded string. The character encoding has already been removed to produce a string of Unicode Code Points. If that's what you have, replace

JSON::decode_json($json_utf8)
JSON::MaybeXS::decode_json($json_utf8)
JSON::PP::decode_json($json_utf8)
JSON::XS::decode_json($json_utf8)
Cpanel::JSON::XS::decode_json($json_utf8)

with

JSON->new->decode($json_ucp)    -or-    JSON::from_json($json_ucp)
JSON::MaybeXS->new->decode($json_ucp)
JSON::PP->new->decode($json_ucp)
JSON::XS->new->decode($json_ucp)
Cpanel::JSON::XS->new->decode($json_ucp)

By the way, unless you want to look at Perl internals, Devel::Peek isn't the right tool for the job. You should be using Data::Dumper or similar instead.

use Data::Dumper qw( Dumper );
# This is the same string as in the OP.
my $value = qq{["\x{41d}\x{434}", "\x{41f}\x{43d}", "\x{412}\x{442}", "\x{421}\x{440}", "\x{427}\x{442}", "\x{41f}\x{442}", "\x{421}\x{431}"]};
local $Data::Dumper::Useqq = 1;
print(Dumper($value));

Output:

$VAR1 = "[\"\x{41d}\x{434}\", \"\x{41f}\x{43d}\", \"\x{412}\x{442}\", \"\x{421}\x{440}\", \"\x{427}\x{442}\", \"\x{41f}\x{442}\", \"\x{421}\x{431}\"]";

Upvotes: 1

Related Questions