CODEWITHSUNDEEP
node.jsfile
andrew
andrew

Reputation: 488

Read nth line in Node.js without reading entire file

I'm trying to use Node.js to get a specific line for a binary search in a 48 Million line file, but I don't want to read the entire file to memory. Is there some function that will let me read, say, line 30 million? I'm looking for something like Python's linecache module.

Update for how this is different: I would like to not read the entire file to memory. The question this is identified as a duplicate of reads the entire file to memory.

Upvotes: 8

Views: 8761

Answers (2)

Boris Chumichev
Boris Chumichev

Reputation: 165

You should use readline module from Node’s standard library. I deal with 30-40 million rows files in my project and this works great.

If you want to do that in a less verbose manner and don’t mind to use third party dependency use nthline package:

const nthline = require('nthline')
    , filePath = '/path/to/100-million-rows-file'
    , rowNumber = 42

nthline(rowNumber, filePath)
  .then(line => console.log(line))

Upvotes: 8

skypjack
skypjack

Reputation: 50540

According to the documentation, you can use fs.createReadStream(path[, options]), where:

options can include start and end values to read a range of bytes from the file instead of the entire file.

Unfortunately, you have to approximate the desired position/line, but it seems to be no seek like function in node js.

EDIT

The above solution works well with lines that have fixed length.

New line character is nothing more than a character like all the others, so looking for new lines is like looking for lines that start with the character a.
Because of that, if you have lines with variable length, the only viable approach is to load them one at a time in memory and discard those in which you are not interested.

Upvotes: 5

Related Questions