CODEWITHSUNDEEP
phpmysqlserializationcontent-management-system
user576820
user576820

Reputation: 1347

How to repair a serialized string which has been corrupted by an incorrect byte count length?

I am using Hotaru CMS with the Image Upload plugin, I get this error if I try to attach an image to a post, otherwise there is no error:

unserialize() [function.unserialize]: Error at offset

The offending code (error points to line with **):

/**
     * Retrieve submission step data
     *
     * @param $key - empty when setting
     * @return bool
     */
    public function loadSubmitData($h, $key = '')
    {
        // delete everything in this table older than 30 minutes:
        $this->deleteTempData($h->db);

        if (!$key) { return false; }

        $cleanKey = preg_replace('/[^a-z0-9]+/','',$key);
        if (strcmp($key,$cleanKey) != 0) {
            return false;
        } else {
            $sql = "SELECT tempdata_value FROM " . TABLE_TEMPDATA . " WHERE tempdata_key = %s ORDER BY tempdata_updatedts DESC LIMIT 1";
            $submitted_data = $h->db->get_var($h->db->prepare($sql, $key));
            **if ($submitted_data) { return unserialize($submitted_data); } else { return false; }** 
        }
    }

Data from the table, notice the end bit has the image info, I am not an expert in PHP so I was wondering what you guys/gals might think?

tempdata_value:

a:10:{s:16:"submit_editorial";b:0;s:15:"submit_orig_url";s:13:"www.bbc.co.uk";s:12:"submit_title";s:14:"No title found";s:14:"submit_content";s:12:"dnfsdkfjdfdf";s:15:"submit_category";i:2;s:11:"submit_tags";s:3:"bbc";s:9:"submit_id";b:0;s:16:"submit_subscribe";i:0;s:15:"submit_comments";s:4:"open";s:5:"image";s:19:"C:fakepath100.jpg";}

Edit: I think I've found the serialize bit...

/**
     * Save submission step data
     *
     * @return bool
     */
    public function saveSubmitData($h)
    {
        // delete everything in this table older than 30 minutes:
        $this->deleteTempData($h->db);

        $sid = preg_replace('/[^a-z0-9]+/i', '', session_id());
        $key = md5(microtime() . $sid . rand());
        $sql = "INSERT INTO " . TABLE_TEMPDATA . " (tempdata_key, tempdata_value, tempdata_updateby) VALUES (%s,%s, %d)";
        $h->db->query($h->db->prepare($sql, $key, serialize($h->vars['submitted_data']), $h->currentUser->id));
        return $key;
    }

Upvotes: 119

Views: 316162

Answers (18)

ButcherFromHell
ButcherFromHell

Reputation: 165

So, we have two scenerios:

1 - already corrupted serialised data stored in SQL (my case) - that needs to be fixed

2 - make sure to not store any corrupted serialised data in the future

I have been there..

1 - couldn't manage with good solution for that.

  • tried functions to recount characters posted above
  • tried that online solution posted above

but the offset error still appears but in different offset. Gave up and finished with sending data one more time and serialise again with 2.

2 - I have tried

$string=base64_encode(serialize($obj));
unserialize(base64_decode($string));

and that did the job when sending data to MySQL one more time.

In my case corrupted data were submitted on webpage in foreign language - with special characters and some different encoding. Something didn't work well with preparing datas to serialise and to store.

Maybe it will help someone.

Upvotes: -1

adilbo
adilbo

Reputation: 970

Quick Fix

Recalculating the length of the elements in serialized array - but don't use (preg_replace) it's deprecated - better use preg_replace_callback:

Edit: New Version now not just wrong length but it also fix line-breaks and count correct characters with aczent (thanks to mickmackusa)

// New Version
$data = preg_replace_callback('!s:\d+:"(.*?)";!s', 
    function($m) {
        return "s:" . strlen($m[1]) . ':"'.$m[1].'";'; 
    }, $data
);

Upvotes: 21

Pham Dai
Pham Dai

Reputation: 11

You can use this for all case:

$newdata = preg_replace_callback(
    '/(?<=^|\{|;)s:(\d+):\"(.*?)\";(?=[asbdiO]\:\d|N;|\}|$)/s',
    function($m){
        return 's:' . strlen($m[2]) . ':"' . $m[2] . '";';
    },
    $badData
);

Upvotes: 1

ATHIRA
ATHIRA

Reputation: 11

change the column size of the particular field(LONGTEXT)

Upvotes: 1

Pardeep Goyal
Pardeep Goyal

Reputation: 164

public function unserializeKeySkills($string) {
    $output = array();
    $string = trim(preg_replace('/\s\s+/', ' ',$string));
    $string = preg_replace_callback('!s:(\d+):"(.*?)";!', function($m) { return 's:'.strlen($m[2]).':"'.$m[2].'";'; }, utf8_encode( trim(preg_replace('/\s\s+/', ' ',$string)) ));
    try {
        $output =  unserialize($string);
    } catch (\Exception $e) {
        \Log::error("unserialize Data : " .print_r($string,true));
    }
    return $output;
}

Upvotes: 5

Mohd Abdul Mujib
Mohd Abdul Mujib

Reputation: 13948

Here is an Online Tool for fixing a corrupted serialized string.

I'd like to add that this mostly happens due to a search and replace done on the DB and the serialization data(specially the key length) doesn't get updated as per the replace and that causes the "corruption".

Nonetheless, The above tool uses the following logic to fix the serialization data (Copied From Here).

function error_correction_serialise($string){
    // at first, check if "fixing" is really needed at all. After that, security checkup.
    if ( unserialize($string) !== true &&  preg_match('/^[aOs]:/', $string) ) {
         $string = preg_replace_callback( '/s\:(\d+)\:\"(.*?)\";/s',    function($matches){return 's:'.strlen($matches[2]).':"'.$matches[2].'";'; },   $string );
    }
    return $string;
}

Upvotes: 3

Даниил Путилин
Даниил Путилин

Reputation: 121

$badData = 'a:2:{i:0;s:16:"as:45:"d";
Is \n";i:1;s:19:"as:45:"d";
Is \r\n";}';

You can not fix a broken serialize string using the proposed regexes:

$data = preg_replace('!s:(\d+):"(.*?)";!e', "'s:'.strlen('$2').':\"$2\";'", $badData);
var_dump(@unserialize($data)); // Output: bool(false)

// or

$data = preg_replace_callback(
    '/s:(\d+):"(.*?)";/',
    function($m){
        return 's:' . strlen($m[2]) . ':"' . $m[2] . '";';
    },
    $badData
);
var_dump(@unserialize($data)); // Output: bool(false)

You can fix broken serialize string using following regex:

$data = preg_replace_callback(
    '/(?<=^|\{|;)s:(\d+):\"(.*?)\";(?=[asbdiO]\:\d|N;|\}|$)/s',
    function($m){
        return 's:' . strlen($m[2]) . ':"' . $m[2] . '";';
    },
    $badData
);

var_dump(@unserialize($data));

Output

array(2) {
  [0] =>
  string(17) "as:45:"d";
Is \n"
  [1] =>
  string(19) "as:45:"d";
Is \r\n"
}

or

array(2) {
  [0] =>
  string(16) "as:45:"d";
Is \n"
  [1] =>
  string(18) "as:45:"d";
Is \r\n"
}

Upvotes: 12

mickmackusa
mickmackusa

Reputation: 48031

The corruption in this question is isolated to a single substring at the end of the serialized string with was probably manually replaced by someone who lazily wanted to update the image filename. This fact will be apparent in my demonstration link below using the OP's posted data -- in short, C:fakepath100.jpg does not have a length of 19, it should be 17.

Since the serialized string corruption is limited to an incorrect byte/character count number, the following will do a fine job of updating the corrupted string with the correct byte count value.

The following regex based replacement will only be effective in remedying byte counts, nothing more.

It looks like many of the earlier posts are just copy-pasting a regex pattern from someone else. There is no reason to capture the potentially corrupted byte count number if it isn't going to be used in the replacement. Also, adding the s pattern modifier is a reasonable inclusion in case a string value contains newlines/line returns.

*For those that are not aware of the treatment of multibyte characters with serializing, you must not use mb_strlen() in the custom callback because it is the byte count that is stored not the character count, see my output...

Code: (Demo with OP's data) (Demo with arbitrary sample data) (Demo with condition replacing)

$corrupted = <<<STRING
a:4:{i:0;s:3:"three";i:1;s:5:"five";i:2;s:2:"newline1
newline2";i:3;s:6:"garçon";}
STRING;

$repaired = preg_replace_callback(
        '/s:\d+:"(.*?)";/s',
        //  ^^^- matched/consumed but not captured because not used in replacement
        function ($m) {
            return "s:" . strlen($m[1]) . ":\"{$m[1]}\";";
        },
        $corrupted
    );

echo $corrupted , "\n" , $repaired;
echo "\n---\n";
var_export(unserialize($repaired));

Output:

a:4:{i:0;s:3:"three";i:1;s:5:"five";i:2;s:2:"newline1
Newline2";i:3;s:6:"garçon";}
a:4:{i:0;s:5:"three";i:1;s:4:"five";i:2;s:17:"newline1
Newline2";i:3;s:7:"garçon";}
---
array (
  0 => 'three',
  1 => 'five',
  2 => 'newline1
Newline2',
  3 => 'garçon',
)

One leg down the rabbit hole... The above works fine even if double quotes occur in a string value, but if a string value contains "; or some other monkeywrenching sbustring, you'll need to go a little further and implement "lookarounds". My new pattern

checks that the leading s is:

  • the start of the entire input string or
  • preceded by ;

and checks that the "; is:

  • at the end of the entire input string or
  • followed by } or
  • followed by a string or integer declaration s: or i:

I haven't test each and every possibility; in fact, I am relatively unfamiliar with all of the possibilities in a serialized string because I never elect to work with serialized data -- always json in modern applications. If there are additional possible leading or trailing characters, leave a comment and I'll extend the lookarounds.

Extended snippet: (Demo)

$corrupted_byte_counts = <<<STRING
a:12:{i:0;s:3:"three";i:1;s:5:"five";i:2;s:2:"newline1
newline2";i:3;s:6:"garçon";i:4;s:111:"double " quote \"escaped";i:5;s:1:"a,comma";i:6;s:9:"a:colon";i:7;s:0:"single 'quote";i:8;s:999:"semi;colon";s:5:"assoc";s:3:"yes";i:9;s:1:"monkey";wrenching doublequote-semicolon";s:3:"s:";s:9:"val s: val";}
STRING;

$repaired = preg_replace_callback(
        '/(?<=^|;)s:\d+:"(.*?)";(?=$|}|[si]:)/s',
        //^^^^^^^^--------------^^^^^^^^^^^^^-- some additional validation
        function ($m) {
            return 's:' . strlen($m[1]) . ":\"{$m[1]}\";";
        },
        $corrupted_byte_counts
    );

echo "corrupted serialized array:\n$corrupted_byte_counts";
echo "\n---\n";
echo "repaired serialized array:\n$repaired";
echo "\n---\n";
print_r(unserialize($repaired));

Output:

corrupted serialized array:
a:12:{i:0;s:3:"three";i:1;s:5:"five";i:2;s:2:"newline1
newline2";i:3;s:6:"garçon";i:4;s:111:"double " quote \"escaped";i:5;s:1:"a,comma";i:6;s:9:"a:colon";i:7;s:0:"single 'quote";i:8;s:999:"semi;colon";s:5:"assoc";s:3:"yes";i:9;s:1:"monkey";wrenching doublequote-semicolon";s:3:"s:";s:9:"val s: val";}
---
repaired serialized array:
a:12:{i:0;s:5:"three";i:1;s:4:"five";i:2;s:17:"newline1
newline2";i:3;s:7:"garçon";i:4;s:24:"double " quote \"escaped";i:5;s:7:"a,comma";i:6;s:7:"a:colon";i:7;s:13:"single 'quote";i:8;s:10:"semi;colon";s:5:"assoc";s:3:"yes";i:9;s:39:"monkey";wrenching doublequote-semicolon";s:2:"s:";s:10:"val s: val";}
---
Array
(
    [0] => three
    [1] => five
    [2] => newline1
newline2
    [3] => garçon
    [4] => double " quote \"escaped
    [5] => a,comma
    [6] => a:colon
    [7] => single 'quote
    [8] => semi;colon
    [assoc] => yes
    [9] => monkey";wrenching doublequote-semicolon
    [s:] => val s: val
)

Upvotes: 3

David
David

Reputation: 6084

After having tried some things on this page without success I had a look in the page-source and remarked that all quotes in the serialized string have been replaced by html-entities. Decoding these entities helps avoiding much headache:

$myVar = html_entity_decode($myVar);

Upvotes: 1

Rajesh Meniya
Rajesh Meniya

Reputation: 813

You can fix broken serialize string using following function, with multibyte character handling.

function repairSerializeString($value)
{

    $regex = '/s:([0-9]+):"(.*?)"/';

    return preg_replace_callback(
        $regex, function($match) {
            return "s:".mb_strlen($match[2]).":\"".$match[2]."\""; 
        },
        $value
    );
}

Upvotes: 3

GarryOne
GarryOne

Reputation: 1488

Another reason of this problem can be column type of "payload" sessions table. If you have huge data on session, a text column wouldn't be enough. You will need MEDIUMTEXT or even LONGTEXT.

Upvotes: 0

Adam Bubela
Adam Bubela

Reputation: 10063

In my case I was storing serialized data in BLOB field of MySQL DB which apparently wasn't big enough to contain the whole value and truncated it. Such a string obviously could not be unserialized.
Once converted that field to MEDIUMBLOB the problem dissipated. Also it may be needed to switch in table options ROW_FORMAT to DYNAMIC or COMPRESSED.

Upvotes: 1

Muayyad Alsadi
Muayyad Alsadi

Reputation: 1593

the official docs says it should return false and set E_NOTICE

but since you got error then the error reporting is set to be triggered by E_NOTICE

here is a fix to allow you detect false returned by unserialize

$old_err=error_reporting(); 
error_reporting($old_err & ~E_NOTICE);
$object = unserialize($serialized_data);
error_reporting($old_err);

you might want to consider use base64 encode/decode

$string=base64_encode(serialize($obj));
unserialize(base64_decode($string));

Upvotes: 3

Ankit Vishwakarma
Ankit Vishwakarma

Reputation: 1691

You will have to alter the collation type to utf8_unicode_ci and the problem will be fixed.

Upvotes: 1

r00tAcc3ss
r00tAcc3ss

Reputation: 1331

I don't have enough reputation to comment, so I hope this is seen by people using the above "correct" answer:

Since php 5.5 the /e modifier in preg_replace() has been deprecated completely and the preg_match above will error out. The php documentation recommends using preg_match_callback in its place.

Please find the following solution as an alternative to the above proposed preg_match.

$fixed_data = preg_replace_callback ( '!s:(\d+):"(.*?)";!', function($match) {      
    return ($match[1] == strlen($match[2])) ? $match[0] : 's:' . strlen($match[2]) . ':"' . $match[2] . '";';
},$bad_data );

Upvotes: 113

Ge Rong
Ge Rong

Reputation: 446

There's another reason unserialize() failed because you improperly put serialized data into the database see Official Explanation here. Since serialize() returns binary data and php variables don't care encoding methods, so that putting it into TEXT, VARCHAR() will cause this error.

Solution: store serialized data into BLOB in your table.

Upvotes: 18

Will
Will

Reputation: 71

This error is caused because your charset is wrong.

Set charset after open tag: 

header('Content-Type: text/html; charset=utf-8');

And set charset utf8 in your database : 

mysql_query("SET NAMES 'utf8'");

Upvotes: 6

Baba
Baba

Reputation: 95161

unserialize() [function.unserialize]: Error at offset was dues to invalid serialization data due to invalid length

Quick Fix

What you can do is is recalculating the length of the elements in serialized array

You current serialized data 

$data = 'a:10:{s:16:"submit_editorial";b:0;s:15:"submit_orig_url";s:13:"www.bbc.co.uk";s:12:"submit_title";s:14:"No title found";s:14:"submit_content";s:12:"dnfsdkfjdfdf";s:15:"submit_category";i:2;s:11:"submit_tags";s:3:"bbc";s:9:"submit_id";b:0;s:16:"submit_subscribe";i:0;s:15:"submit_comments";s:4:"open";s:5:"image";s:19:"C:fakepath100.jpg";}';

Example without recalculation 

var_dump(unserialize($data));

Output 

Notice: unserialize() [function.unserialize]: Error at offset 337 of 338 bytes

Recalculating 

$data = preg_replace('!s:(\d+):"(.*?)";!e', "'s:'.strlen('$2').':\"$2\";'", $data);
var_dump(unserialize($data));

Output

array
  'submit_editorial' => boolean false
  'submit_orig_url' => string 'www.bbc.co.uk' (length=13)
  'submit_title' => string 'No title found' (length=14)
  'submit_content' => string 'dnfsdkfjdfdf' (length=12)
  'submit_category' => int 2
  'submit_tags' => string 'bbc' (length=3)
  'submit_id' => boolean false
  'submit_subscribe' => int 0
  'submit_comments' => string 'open' (length=4)
  'image' => string 'C:fakepath100.jpg' (length=17)

Recommendation .. I

Instead of using this kind of quick fix ... i"ll advice you update the question with

  • How you are serializing your data

  • How you are Saving it ..

================================ EDIT 1 ===============================

The Error

The Error was generated because of use of double quote " instead single quote ' that is why C:\fakepath\100.png was converted to C:fakepath100.jpg

To fix the error

You need to change $h->vars['submitted_data'] From (Note the singe quite ' )

Replace

 $h->vars['submitted_data']['image'] = "C:\fakepath\100.png" ;

With

 $h->vars['submitted_data']['image'] = 'C:\fakepath\100.png' ;

Additional Filter

You can also add this simple filter before you call serialize 

function satitize(&$value, $key)
{
    $value = addslashes($value);
}

array_walk($h->vars['submitted_data'], "satitize");

If you have UTF Characters you can also run 

 $h->vars['submitted_data'] = array_map("utf8_encode",$h->vars['submitted_data']);

How to detect the problem in future serialized data 

  findSerializeError ( $data1 ) ;

Output

Diffrence 9 != 7
    -> ORD number 57 != 55
    -> Line Number = 315
    -> Section Data1  = pen";s:5:"image";s:19:"C:fakepath100.jpg
    -> Section Data2  = pen";s:5:"image";s:17:"C:fakepath100.jpg
                                            ^------- The Error (Element Length)

findSerializeError Function 

function findSerializeError($data1) {
    echo "<pre>";
    $data2 = preg_replace ( '!s:(\d+):"(.*?)";!e', "'s:'.strlen('$2').':\"$2\";'",$data1 );
    $max = (strlen ( $data1 ) > strlen ( $data2 )) ? strlen ( $data1 ) : strlen ( $data2 );

    echo $data1 . PHP_EOL;
    echo $data2 . PHP_EOL;

    for($i = 0; $i < $max; $i ++) {

        if (@$data1 {$i} !== @$data2 {$i}) {

            echo "Diffrence ", @$data1 {$i}, " != ", @$data2 {$i}, PHP_EOL;
            echo "\t-> ORD number ", ord ( @$data1 {$i} ), " != ", ord ( @$data2 {$i} ), PHP_EOL;
            echo "\t-> Line Number = $i" . PHP_EOL;

            $start = ($i - 20);
            $start = ($start < 0) ? 0 : $start;
            $length = 40;

            $point = $max - $i;
            if ($point < 20) {
                $rlength = 1;
                $rpoint = - $point;
            } else {
                $rpoint = $length - 20;
                $rlength = 1;
            }

            echo "\t-> Section Data1  = ", substr_replace ( substr ( $data1, $start, $length ), "<b style=\"color:green\">{$data1 {$i}}</b>", $rpoint, $rlength ), PHP_EOL;
            echo "\t-> Section Data2  = ", substr_replace ( substr ( $data2, $start, $length ), "<b style=\"color:red\">{$data2 {$i}}</b>", $rpoint, $rlength ), PHP_EOL;
        }

    }

}

A better way to save to Database 

$toDatabse = base64_encode(serialize($data));  // Save to database
$fromDatabase = unserialize(base64_decode($data)); //Getting Save Format

Upvotes: 238

Related Questions