CODEWITHSUNDEEP
node.js
aashu1807
aashu1807

Reputation: 202

how to scrape data from a website using node

I am new in Node JS. from morning i am trying to scrape data from website https://www.pmkisan.gov.in/StateDist_Beneficiery.aspx

[![enter image description here][1]][1]

I want to store that date also in db table. Currently i am putting hard coded date. But how can i scrape that date?

my code

(async () => {
    
    

    await performDbActions(webData);

})();


async function performDbActions(data) {
    let dataToBeInsert = {};
    // console.log(data);
    for (const d of data) {
        if (Object.keys(d).length) {
            // console.log(d);
            let district = await db.sequelize.query("select * from abc where name=:districtName or other_names like :districtLike", {
                replacements: {districtName: d['name'], districtLike: '%' + d['name'] + '%'},
                raw: true,
                type: db.sequelize.QueryTypes.SELECT
            });
            
                    delete d['sno'];
                    delete d['name'];
           
                    d['as_on'] = '2020-02-06';
                
                
            }
        }
    }
   
}

Upvotes: 1

Views: 94

Answers (1)

eol
eol

Reputation: 24565

According to the page's source code the date you're looking for is inside a <span> that has the id ContentPlaceHolder1_lbldate. So you can just use cheerio to get its text-content and pass the result to performDbActions as an additional parameter:

//...
const date = $('#ContentPlaceHolder1_lbldate').text();
//...
await performDbActions(webData, date);
// ...

async function performDbActions(data, date) {
  // ...
  // it would be safer to use an external date-library like moment.js but here's a way to convert the date in plain js 
  const dateParts =date.split('/');
  const dateObj = new Date(dateParts[2], dateParts[1] - 1, dateParts[0]);
  d['created_at'] = dateObj;
}

Note that the date is in format dd/mm/yyyy, so you may have to convert it to your desired format.

Upvotes: 1

Related Questions