CODEWITHSUNDEEP
node.js
user12341234
user12341234

Reputation: 1487

Nodejs exec child process stdout not getting all the chunks

I'm trying to send messages from my child process to main my process but some chunks are not being sent, possibly because the file is too big.

main process:

let response = ''
let error = ''
await new Promise(resolve => {
  const p = exec(command)
  p.stdout.on('data', data => {
    // this gets triggered many times because the html string is big and gets split up
    response += data
  })
  p.stderr.on('data', data => {
    error += data
  })
  p.on('exit', resolve)
})
console.log(response)

child process:

// only fetch 1 page, then quit
const bigHtmlString = await fetchHtmlString(url)
process.stdout.write(bigHtmlString)

I know the child process works because when I run the it directly, I can see the end of the file in in the console. But when I run the main process, I cannot see the end of the file. It's quite big so I'm not sure exactly what chunks are missing.

edit: there's also a new unknown problem. when I add a wait at the end of my child process, it doesn't wait, it closes. So I'm guessing it crashes somehow? I'm not seeing any error even with p.on('error', console.log) example:

const bigHtmlString = await fetchHtmlString(url)
process.stdout.write(bigHtmlString)
// this never gets executed, the process closes. The wait works if I launch the child process directly
await new Promise(resolve => setTimeout(resolve, 1000000))

Upvotes: 0

Views: 1782

Answers (2)

Sebastian Kaczmarek
Sebastian Kaczmarek

Reputation: 8515

My suggestion from the comments resolved the issue so I'm posting it as an answer.

I would suggest using spawn instead of exec. The latter buffers the output and flushes it when the process is ended (or the buffer is full) while spawn is streaming the output which is better for huge output like in your case

Upvotes: 1

Tudor Constantin
Tudor Constantin

Reputation: 26861

process.stdout.write(...) returns true/false depending on whether it wrote the string or not. If it returns false, you can listen() to the drain event to make sure it finishes.

Something like this:

const bigHtmlString = await fetchHtmlString(url);
const wrote = process.stdout.write(bigHtmlString);

if (!wrote){
  // this effectively means "wait for this
  // event to fire", but it doesn't block everything
  process.stdout.on('drain', ...doSomethingHere)
}

Upvotes: 2

Related Questions