CODEWITHSUNDEEP
javascriptnode.jstypescriptelectronexceljs
MusicDev
MusicDev

Reputation: 94

How to read large files (Excel) in NodeJS without pausing UI

I'm currently using ExcelJS to read large Excel files (10,000+ lines) into a NodeJS/Angular/Electron app. It reads the smaller files just fine, but the larger files take anywhere from 3.9 to 5 seconds to read, and during that time, the CSS doesn't update at all.

I'm currently using async/await to load the files, as I thought that would allow other operations to happen while waiting on the file load, as I thought I/O operations in Node had their own thread. I also read that CPU intensive tasks block all other Node processes.

This is the code that currently loads the workbook:

async openWorkbook(filename: string) {
    this.loading = true;
    const workbook = new Workbook();
    const promise = workbook.xlsx.readFile(this.path + '/sheets/' + filename)
      .then(() => {
        // use workbook
        workbook.getWorksheet(1).name = workbook.getWorksheet(1).name.slice(0, 31);
        const excelFile: ExcelFile = { workbook: workbook, filename: filename };
        this.wbSource.next(excelFile);
      });

    const read = await promise;
  }

This code works the same as the callback implementation, they both block the app's CSS. How would I go about reading a file (or do any CPU intensive task) without blocking the UI?

Upvotes: 1

Views: 12898

Answers (3)

Estradiaz
Estradiaz

Reputation: 3563

The first thing i would do in this case is preventing memory overhead by not "opening" large files at all.

So something like this should work:

const stream = fs.createReadStream(filePath);
const workbook = new Excel.Workbook();
stream.pipe(workbook.xlsx.createInputStream());

and as those chunks block the cycle for a minor amount of time you can use this even in the javascript event loop ;)

Upvotes: 3

I wrestled a bear once.
I wrestled a bear once.

Reputation: 23379

I'm just going to leave my solution here using my Thread function from the comments.

async openWorkbook(filename: string) {
  this.loading = true;

  var path = this.path + '/sheets/' + filename;
  const excelFile = await Thread(path, function(path, exit) {

    // You'll need to import excel again since it's a new thread
    import Excel from 'exceljs';

    const workbook = new Excel.Workbook();
    workbook.xlsx.readFile(path).then(() => {
      workbook.getWorksheet(1).name = workbook.getWorksheet(1).name.slice(0, 31);
      const excelFile: ExcelFile = {
        workbook: workbook,
        filename: filename
      };
      exit(excelFile);
    });
  });

  this.wbSource.next(excelFile);

}

Upvotes: 0

vatz88
vatz88

Reputation: 2452

JavaScript is a single threaded language. If one of your task is taking long to complete, other tasks like rendering will be blocked. I can't write complete code for you but Web Worker seems a perfect solution for what you're trying to do. (Which people have already suggested in comments)

You can refer:

https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers

Additionally, also check out https://github.com/GoogleChromeLabs/comlink which is a wrapper around the Web Worker APIs. It'll make it easier for you to migrate existing code to worker with less hassle.

Upvotes: 1

Related Questions