CODEWITHSUNDEEP
phpmysqlrss
Stephanie Kostova
Stephanie Kostova

Reputation: 88

How to avoid duplicate results when import from RSS

I'm importing data from RSS to mysql using core php, every x hours, but I'm struggling with duplicate entries.

$rss_url = 'https://promograd.bg/feed/agg/common.xml?a=143';
    $xml = simplexml_load_file($rss_url);
    foreach($xml->ITEM as $item) {
        $title = mysqli_real_escape_string($link, $item->TITLE);
        $offerUrl = $item->URL;
        $description = mysqli_real_escape_string($link, $item->DESCRIPTION);
        $offerTerms = mysqli_real_escape_string($link, $item->TERMS);
        $originalPrice = $item->ORIGINAL_PRICE;
        $finalPrice = $item->FINAL_PRICE;
        $offerDiscount = $item->DISCOUNT;
        $offerSales = $item->SALES;
        $offerEnds = $item->DEAL_END;
        $lat_coordinates = $item->LAT;
        $lng_coordinates = $item->LNG;
        $city = mysqli_real_escape_string($link, $item->CITY);
        $category = mysqli_real_escape_string($link, $item->CATEGORY);

        $img = $item->IMAGE;

        $query = mysqli_query($link, "
        INSERT INTO......       
        }

My issue is when I run this script it will import the same results, with not much new.. How I can avoid duplicated results?

Upvotes: 0

Views: 613

Answers (2)

SachinPatil4991
SachinPatil4991

Reputation: 774

Put a unique key on table for column which you do not want to duplicated. Or you can put unique key on multiple columns like combination of title and url as well.

Now in inert query

use insert ignore to avoid insert duplicate entries

or use on duplicate key update to update some fields when duplicate entry found. like if you want to update new price for same existing record.

Upvotes: 0

Rakesh Jakhar
Rakesh Jakhar

Reputation: 6388

For example, if you are checking the title for duplicate you can try this:-

$rss_url = 'https://promograd.bg/feed/agg/common.xml?a=143';
$xml = simplexml_load_file($rss_url);
$tempRecords = array(); // temp array store titles
foreach($xml->ITEM as $item) {
    $title = mysqli_real_escape_string($link, $item->TITLE);
    if(in_array($title, $tempRecords)){ //skip if exists
        continue;
    }else{ // else insert
        //$title = mysqli_real_escape_string($link, $item->TITLE);
        $tempRecords[] = $title; //assign to temp array

        $offerUrl = $item->URL;
        $description = mysqli_real_escape_string($link, $item->DESCRIPTION);
        $offerTerms = mysqli_real_escape_string($link, $item->TERMS);
        $originalPrice = $item->ORIGINAL_PRICE;
        $finalPrice = $item->FINAL_PRICE;
        $offerDiscount = $item->DISCOUNT;
        $offerSales = $item->SALES;
        $offerEnds = $item->DEAL_END;
        $lat_coordinates = $item->LAT;
        $lng_coordinates = $item->LNG;
        $city = mysqli_real_escape_string($link, $item->CITY);
        $category = mysqli_real_escape_string($link, $item->CATEGORY);

        $img = $item->IMAGE;

        $query = mysqli_query($link, "
        INSERT INTO......  
    }

    }

You can also do it using the mysql query, please refer the link

https://ypereirareis.github.io/blog/2016/03/22/mysql-insert-ignore-alternatives/

Upvotes: 1

Related Questions