CODEWITHSUNDEEP
javascriptnode.jsarraysobject
user11368144
user11368144

Reputation:

How to merge arrays that are passed into the function?

So I've been working on my project that involves scraper.

So workflow is next: There are two scrapers right now. Data is being parsed and pushed into array for each individual scraper and passed to the merge component.

So the merge component looks like this:

let mergedApartments = []; //Creating merged list of apartments
exports.mergeData = (apartments) => {
  //Fetching all apartments that are passed from scraper(s)
  mergedApartments.push(...apartments); //Pushing apartments into the list
  console.log(mergedApartments.length);
};

So right now output of mergedApartments.length is 9 39. So the first function that calls mergeData() and pass it an array have 9 objects inside it, and the other scraper have 30 objects inside it's array, who is again passed to the mergeData.

Now this is not what I've expected. I've expected one array with all merged objects from the scrapers. Right now, scraperno1 send apartments and it's added to the mergedApartments, then scraperno2 sends apartments and it's overwriting that array by adding new apartments objects into the array.

Now I want different output: I just want to get one list with all merged objects from the arrays. Because this data will be passed to the storing component, and I don't want to query DB multiple times, because for each new mergedApartments list, data will be inserted and creating duplicate values - throwing an error.

So what I've tried: I've tried creating some kind of a counter which counts number of time that function mergeData is called, and then do the logic about merging but no success.

So I just want my array to have one output of mergedApartments.length - in this case 39.

Thanks!

EDIT Here how one of the scraper looks:

const merge = require('../data-functions/mergeData');
const axios = require('axios');
const cheerio = require('cheerio');

//function for olx.ba scraper. Fetching raw html data and pushing it into array of objects. Passing data to merge function

exports.santScraper = (count) => {
  const url = `https://www.sant.ba/pretraga/prodaja-1/tip-2/cijena_min-20000/stranica-${count}`;

  const santScrapedData = [];
  const getRawData = async () => {
    try {
      await axios.get(url).then((response) => {
        const $ = cheerio.load(response.data);

        $('div[class="col-xxs-12 col-xss-6 col-xs-6 col-sm-6 col-lg-4"]').each(
          (index, element) => {
            const getLink = $(element).find('a[class="re-image"]').attr('href');
            const getDescription = $(element).find('a[class="title"]').text();
            const getPrice = $(element)
              .find('div[class="prices"] > h3[class="price"]')
              .text()
              .replace(/\.| ?KM$/g, '')
              .replace(',', '.');
            const getPicture = $(element).find('img').attr('data-original');
            const getSquaremeters = $(element)
              .find('span[class="infoCount"]')
              .first()
              .text()
              .replace(',', '.')
              .split('m')[0];
            const pricepersquaremeter =
              parseFloat(getPrice) / parseFloat(getSquaremeters);

            santScrapedData[index] = {
              id: getLink.substring(42, 46),
              link: getLink,
              descr: getDescription,
              price: Math.round(getPrice),
              pictures: getPicture,
              sqm: Math.round(getSquaremeters),
              ppm2: Math.round(pricepersquaremeter),
            };
          }
        );
        merge.mergeData(santScrapedData); //here i'm calling function and passing array to function
      });
    } catch (error) {
      console.log(error);
    }
  };
  getRawData();
};

Other scraper looks the same(it's same calling of the function)

Upvotes: 0

Views: 76

Answers (1)

Mike Malyi
Mike Malyi

Reputation: 1101

For this, you need to use concat function from the Array prototype

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Array/concat

exports.mergeData = (apartments) => {
  mergedApartment = mergedApartments.concat(apartments);
};

exports.sendData = () => {
  console.log(mergedApartment.length);
}

and in your main script

getRawData().then(merge.sendData);

Upvotes: 2

Related Questions