CODEWITHSUNDEEP
javascriptnode.jsperformanceio
Hello Mellow
Hello Mellow

Reputation: 169

Writing to file multiple times vs Saving it in memory and writing once

I hope it's okay to ask questions about optimization here, if not let me know where I can and i'll delete this post.

Also if it matters, I'm using NodeJS.

So I'm coming up to a point in time where I want to optimize my application. I'm still in the process of creating it, but I want to think long term from the beginning. With that being said, I have two optimization questions

1) In one of my functions I have an array, I loop through this array and I create some text and write/append it to the file through each loop. Meaning if there are 5 items in the array, I'd write to the file 5 times. However on the other hand I could create a variable, through each loop append the text to the variable, and at the very end write to the file. The text in question/length of the array could be few to potentially unlimited (but learning much more on the side of the few)

2) What's faster, deleting a file or writing to the file an empty string (when I write to the file I use appendFile so it appends to it if the file exists or creates the file if it doesn't, so either deleting the file or writing to the file works)

Thanks!

On the topic of optimization, I'm using a custom scroll wheel for one of my elements and for a brief few seconds when I refresh the page the default scroll wheel appears and then the custom one loads, the custom one is smaller in size so it flickers the elements position when it switches. Is there a way to like not load the element till the scroll wheel css file is loaded or something along those lines?

Upvotes: 1

Views: 1197

Answers (1)

Manuel Spigolon
Manuel Spigolon

Reputation: 12870

Generally, there are 3 main ways to do that with pro/cons:

  • append to the file: open and close always the file descriptor is slow, but you are sure of what you have in file system
  • stream to file: open the file and write to it, close when necessary
  • buffer and write once: beware on program exit (SIGNT), all the buffer could be lost if you don't manage it correctly

So it depends on your scope: speed? robustness? high rate changes on a file?

After your consideration, to find which one is fastest you can write a benchmark like this:

const Benchmark = require('benchmark');
const suite = new Benchmark.Suite;


const { Readable } = require('stream')

const util = require('util');
const fs = require('fs');

const append = util.promisify(fs.appendFile);


const text = new Array(500).fill('a'.repeat(5000))

// add tests
const banch = suite
  .add('Append sequentially', {
    defer: true,
    fn: function (deferred) {
      Promise.all(text.map(t => append('append.txt', `${t}\n`)))
        .then(() => deferred.resolve())
    }
  })
  .add('Write once', {
    defer: true,
    fn: function (deferred) {
      const out = text.reduce((a, b) => `${a}\n${b}`, '')
      fs.writeFile('write.txt', out, () => deferred.resolve())
    }
  })
  .add('Stream', {
    defer: true,
    fn: function (deferred) {
      const readable = new Readable()
      const writerStream = fs.createWriteStream('stream.txt')
        .on('finish', () => deferred.resolve())
      readable.pipe(writerStream);
      text.forEach(s => readable.push(`${s}\n`))
      readable.push(null)
    }
  })
  .on('cycle', function (event) {
    console.log(String(event.target));
  })
  .on('complete', function () {
    console.log('Fastest is ' + this.filter('fastest').map('name'));
  })

banch.run({ async: true })

That will output (on my pc):

Append sequentially x 8.69 ops/sec ±21.07% (45 runs sampled)

Write once x 52.22 ops/sec ±5.27% (63 runs sampled)

Stream x 37.76 ops/sec ±2.72% (63 runs sampled)

Fastest is Write once

with node 8

Upvotes: 3

Related Questions