CODEWITHSUNDEEP
node.jsexpressasynchronouscallbackmiddleware
Luke
Luke

Reputation: 5708

Stop function from being invoked multiple times

I'm in the process of building a file upload component that allows you to pause/resume file uploads.

The standard way to achieve this seems to be to break the file into chunks on the client machine, then send the chunks along with book-keeping information up to the server which can store the chunks into a staging directory, then merge them together when it has received all of the chunks. So, this is what I am doing.

I am using node/express and I'm able to get the files fine, but I'm running into an issue because my merge_chunks function is being invoked multiple times.

Here's my call stack:

router.post('/api/videos', 
    upload.single('file'), 
    validate_params, 
    rename_uploaded_chunk,
    check_completion_status,
    merge_chunks,
    record_upload_date,
    videos.update,
    send_completion_notice
);

the check_completion_status function is implemented as follows:

/* Recursively check to see if we have every chunk of a file */
var check_completion_status = function (req, res, next) {
  var current_chunk = 1;
  var see_if_chunks_exist = function () {
    fs.exists(get_chunk_file_name(current_chunk, req.file_id), function (exists) {
      if (current_chunk > req.total_chunks) { 
        next(); 
      } else if (exists) {
        current_chunk ++;
        see_if_chunks_exist();
      } else { 
        res.sendStatus(202);
      } 
    });
  };
  see_if_chunks_exist();
};

The file names in the staging directory have the chunk numbers embedded in them, so the idea is to see if we have a file for every chunk number. The function should only next() one time for a given (complete) file.

However, my merge_chunks function is being invoked multiple times. (usually between 1 and 4) Logging does reveal that it's only invoked after I've received all of the chunks.

With this in mind, my assumption here is that it's the async nature of the fs.exists function that's causing the issue.

Even though the n'th invocation of check_completion_status may occur before I have all of the chunks, by the time we get to the nth call to fs.exists(), x more chunks may have arrived and been processed concurrently, so the function can keep going and in some cases get to the end and next(). However those chunks that arrived concurrently are also going to correspond to invocations of check_completion_status, which are also going to next() because we obviously have all of the files at this point.

This is causing issues because I didn't account for this when I wrote merge_chunks.

For completeness, here's the merge_chunks function:

var merge_chunks = (function () {

  var pipe_chunks = function (args) {
    args.chunk_number = args.chunk_number || 1;
    if (args.chunk_number > args.total_chunks) { 
      args.write_stream.end();
      args.next(); 
    } else {
      var file_name = get_chunk_file_name(args.chunk_number, args.file_id)
      var read_stream = fs.createReadStream(file_name);
      read_stream.pipe(args.write_stream, {end: false});
      read_stream.on('end', function () {
        //once we're done with the chunk we can delete it and move on to the next one.
        fs.unlink(file_name);
        args.chunk_number += 1;
        pipe_chunks(args);
      }); 
    }  
  };

  return function (req, res, next) {
    var out = path.resolve('videos', req.video_id);
    var write_stream = fs.createWriteStream(out);
    pipe_chunks({
      write_stream: write_stream,
      file_id: req.file_id,
      total_chunks: req.total_chunks,
      next: next
    });
  };

}());

Currently, I'm receiving an error because the second invocation of the function is trying to read the chunks that have already been deleted by the first invocation.

What is the typical pattern for handling this type of situation? I'd like to avoid a stateful architecture if possible. Is it possible to cancel pending handlers right before calling next() in check_completion_status?

Upvotes: 1

Views: 74

Answers (1)

Yuri Zarubin
Yuri Zarubin

Reputation: 11677

If you just want to make it work ASAP, I would use a lock (much like a db lock) to lock the resource so that only one of the requests processes the chunks. Simply create a unique id on the client, and send it along with the chunks. Then just store that unique id in some sort of a data structure, and look that id up prior to processing. The example below is by far not optimal (in fact this map will keep growing, which is bad), but it should demonstrate the concept

// Create a map (an array would work too) and keep track of the video ids that were processed. This map will persist through each request.
var processedVideos = {};

var check_completion_status = function (req, res, next) {
  var current_chunk = 1;
  var see_if_chunks_exist = function () {
    fs.exists(get_chunk_file_name(current_chunk, req.file_id), function (exists) {
      if (processedVideos[req.query.uniqueVideoId]){
        res.sendStatus(202);
      } else if (current_chunk > req.total_chunks) { 
        processedVideos[req.query.uniqueVideoId] = true;
        next(); 
      } else if (exists) {
        current_chunk ++;
        see_if_chunks_exist();
      } else { 
        res.sendStatus(202);
      } 
    });
  };
  see_if_chunks_exist();
};

Upvotes: 1

Related Questions