CODEWITHSUNDEEP
phpwordpress
warren
warren

Reputation: 61

disable duplicate post check in wp-all-import wordpress plugin

I am using the wp all import wordpress plugin to import posts to my wordpress blog from a csv file. The file has over 10000 posts, the problem is the plugin posting gets slower as the database size increases or if it posts about 1000 posts the number of posts published decreases to 1/4th of the rate when the process began.

Is it possible to prevent duplicate check by the plugin?

Or can I limit the data read from the wordpress database to 100 posts instead of reading the 1000 posts?

Upvotes: 5

Views: 11678

Answers (3)

Brev Tiw
Brev Tiw

Reputation: 602

There is actually a filter that does this now. This way it should stick when the plugin updates. You will not have to use a 'Hack' as described in the current selected answer. 

apply_filters('wp_all_import_is_check_duplicates', true, $this->id);

So you can write a function like this to prevent the checking for duplicate records. Add it to your functions.php file or to your plugin.

//My Custom FILTER to Disable checking for duplicates
add_filter('wp_all_import_is_check_duplicates', 'disable_check_dupe_func', 10,     2);

//My Custom FUNCTION to Disable checking for duplicates
function disable_check_dupe_func($truefalse, $thisid){
    $truefalse = false;
    return $truefalse;
}//disable_check_dupe_func($truefalse, $thisid){

I am using the Pro version, so if this filter is not on the Free version, I will change the answer.

Here are my results. My file had 100,000 records in it. I had no need to check for duplicates. Make sure you understand the implications of this before apply this filter.

Before applying filter: 18 minutes 36 seconds - 25% ~25,000 records - I canceled it on purpose.

enter image description here

After applying filter: 10 minutes 37 seconds - 27% ~27,000 records

10 minutes 37 seconds - 27% ~27,000 records

Total Runtime ended up being: 36 minutes 23 seconds - 100% 100,000 records. enter image description here

So for me it saved a lot of time.

Upvotes: 6

enigma
enigma

Reputation: 3491

The problem

This slowing down over time is a known issue, and an explanation is documented on the plugin website, with the most relevant parts reproduced below.

Splitting a file into chunks is a useful way to reduce the amount WP All Import slows down by at the end of the import process.

The reason for the slowdown is partially because WP All Import must read further into the file later in the import process. On each iteration, the “file pointer” is reset.

[...] During late iterations, for example, when importing records 15000 – 16000, WP All Import must read 15000 records into the file before it can even start importing the data.

Splitting the file into chunks prevents this slowdown – but at a cost – it must temporarily create many new files on your server.

[...] It is not possible to completely eliminate slowdown, because as your database grows larger, your server and WP All Import must do more work.

with the example you mentioned about duplicates.

Solutions

do_action

As mentioned above, splitting the import file into chunks will help alleviate the slowdown.

The linked page also describes that the plugin has been designed to operate without calling do_action, and it is recommended that this is disabled, so long as no other plugins or processes need to use it.

...we created a version of this function without any of the do_action calls, and checking this box will make WP All Import use our customized wp_insert_post function instead of the default provided by WordPress.

The checkbox mentioned is available under the heading "Configure Advanced Settings" in step 4 of the import.

Disabling duplicate post checking

I have been unable to find a settings value to do this, so have had to resort to editing the plugin's source code to accomplish this.

Open up the file /wp-content/plugins/wp-all-import/models/import/record.php and do a search for "// if Auto Matching re-import option selected", which should be line 739, and enter /* to start a block comment from the line above it. Then scroll down to line 807 (just above "// Duplicate record is founded") and enter */ to end the block comment. This will prevent the import process from checking for duplicate posts, and should speed it up considerably with large imports. This has been verified to not cause any errors on the latest release of the free plugin (apart from obvious duplications should they arise), but I do not have a dataset large enough to test any reliable performance increases.

It is worth noting that this is a "hack", the changes made will be reverted on any updates to the plugin, and it should not be considered a stable fix.

Upvotes: 2

Harry Geo
Harry Geo

Reputation: 1173

it seems your server is struggling as this operation is resource intensive.Sending all these SQL queries in your DB will surely slow things down in your server. You could try some things and see if it helps.

  • Try to split the script into chunks and parse a small number of posts at the time (ex. 1000).
  • Disable do_action
  • Try to increase the execution time of php scripts (this is unlikely to help, but if nothing helps give it a try).
  • Import an sql file backup directly into your database using a DBMS (Mysql Workbench, PgAdmin etc...), and import the xml file that maps the db with the posts. More info:

About WP all import here

About Wordpress backups here

Upvotes: 8

Related Questions