CODEWITHSUNDEEP
node.jsstreamfs
user1513388
user1513388

Reputation: 7451

Reading and parsing large files

I have very large file which I need to parse and read the data between the "BEGIN DATA" and "END DATA" delimiters, then do something like decoding the block.

I can open the file easily using the "fs" library like so:

  fs.readFile(files[0], 'utf8', function (err, data) {
  if (err) return console.log(err);
  console.log(data)
 });

However, I need to then read the data between the delimiters in blocks via a stream so not use large amounts of memory.

        -----BEGIN DATA-----
        MIIEzDCCArSgAwIBAgIVCugKYzMN5ra8zPWxYE8pUU9SxjYSMA0GCSqGSIb3DQEB
        CwUAMHAxCzAJBgNVBAYTAkdCMRUwEwYDVQQIDAxXYXJ3aWNrc2hpcmUxEDAOBgNV
        BAcMB1dhcndpY2sxEDAOBgNVBAoMB0VudHJ1c3QxETAPBgNVBAsMCFBLSSBURUFN
        -----END DATA-----
        -----BEGIN DATA-----
        MIIETzCCAjegAwIBAgIVBShP2Mx74DZEyNKwYZZPGntRmSWnMA0GCSqGSIb3DQEB
        DQUAMHIxCzAJBgNVBAYTAkdCMRUwEwYDVQQIDAxXYXJ3aWNrc2hpcmUxEDAOBgNV
        BAcMB1dhcndpY2sxDDAKBgNVBAoMA0lCTTERMA8GA1UECwwIUEtJIFRFQU0xGTAX
        5/62
        -----END DATA-----

Upvotes: 0

Views: 449

Answers (1)

dape
dape

Reputation: 162

Easiest way is to use a stream library coupled to node's fs.createReadStream, in your case the splitBy method in Highland.js would be suitable:

_(fs.createReadStream(files[0], { encoding: 'utf8' }))
  .splitBy('-----BEGIN DATA-----')
  .splitBy('-----END DATA-----')
  .each(_.log)

Upvotes: 1

Related Questions