CODEWITHSUNDEEP
javascriptnode.jsgulp
Havoc P
Havoc P

Reputation: 8467

Parsing multiple files to generate one file in gulp

In gulp, I have multiple files that I parse into arrays of objects; then I want to merge all those arrays into one, and output a single file based on the combined array.

file1 parsed into array1
file2 parsed into array2
file3 parsed into array3
combinedArray = array1 + array2 + array3
combinedArray used to generate outputFile

I'm not sure how to combine the three arrays into one array using gulp - I guess I want to reduce or fold the stream elements.

Given these functions, how would you write this?

I'm sure this is simple but I'm not guessing the right search terms to find the answer. Not very familiar with node...

A contrived example of the problem would be to have several files full of names; extract names starting with "B" from each file; then output an alphabetized, de-duplicated list of names starting with "B." So it isn't just concatenation of the files, it's parsing into a data structure and combining the data structures into one, then outputting a file based on the combination.

Upvotes: 2

Views: 709

Answers (2)

Nick Perkins
Nick Perkins

Reputation: 8284

You have to get into some interesting plug-ins... I have implemented your "B-names" requirements below:

In the first step, I transform the contents of each input file into a JSON string that has all the "B" names as keys and "true" as values ( works like a Set, eliminates duplicates ).

In the second step, I "reduce" all the files into one file. ( this is bit like the "reduce" in functional programming ). The contents of each file is transformed from JSON back into an object, and the keys are all merged into one final object ( again eliminating duplicates between files ). The final step in the "reduce" pulls they keys from the final object, sorts them, and puts them all into a single file called "b_names".

(This code is Coffeescript)

gulp = require 'gulp'
transform  = (require 'gulp-insert').transform
reduce_file = require 'gulp-reduce-file'

gulp.task 'parse_and_merge', ->
    gulp.src 'names*', cwd:'./input'
        .pipe transform (contents,file)->
            b_names = {}
            for b_name in contents.match /B\w+/igm
                b_names[b_name] = true
            return JSON.stringify b_names
        .pipe gulp.dest 'output' # not required, but maybe informative
        .pipe reduce_file 'b_names',
        (file,memo)->
            this_nameset = JSON.parse(file.contents)
            for key,val of this_nameset
                memo[key] = true
            return memo
        ,(memo)->
            b_names = Object.keys(memo)
            b_names.sort()
            return b_names.join('\n')
        ,{}
        .pipe gulp.dest 'output'



names1:
  John
  Bob
  Charles

names2:
  Joe
  Buster
  Michael

names3:
  Jacob
  Barack
  Keith
  Bob
  Bob

b_names:
  Barack
  Bob
  Buster

Upvotes: 1

Havoc P
Havoc P

Reputation: 8467

Here's what I came up with so far, though if anyone can provide a better answer I'm happy to accept it instead. Thanks Arnelle Balane for the pointer to event-stream.

es = require "event-stream"
reduce = require "stream-reduce"

asyncParseFile = (file, callback) ->
  stuff = parseFile(file)
  callback(null, stuff)

asyncCreateFile = (stuff, callback) ->
  file = createFile(stuff)
  callback(null, file)

gulp.src myfiles
 .pipe(es.map(asyncParseFile))
 .pipe(reduce(arrayConcat, []))
 .pipe(es.map(asyncCreateFile))
 .pipe(savefile())

Upvotes: 0

Related Questions