CODEWITHSUNDEEP
javascriptd3.js
Juan Sebastian Silva
Juan Sebastian Silva

Reputation: 45

Load and parse multiple csv files with D3.js

I'm implementing a map visualization using D3.js. I have a number of x csv files (matrices). I want to load them and sum all values from all files.

I implemented a for loop over the array of names and load and parse the data using d3.text, but due to async behavior, I can't find a way to do this (I get undefined from console.log first and then the result of the for loop).

I've tried to use async.js but I cant figure out how to allow the flow of the for loop. Here is my code:

var total=[];
var matrix=[];

for(var i=0; i<filenames.length; i++){

    d3.text(filenames[i], function(text){
      matrix = d3.csv.parseRows(text).map(function (row){
        return row.map(function(value){ 
          return +value;
        });
      });
    });

    //sum the matrix pseudocode
    for(...){
      total = total + matrix;
    }
}

//here I need the sum of all files and then do other stuffs with the total:
console.log(total);
...
...

How can I archieve this? Thanks.

Upvotes: 1

Views: 1837

Answers (1)

jhinzmann
jhinzmann

Reputation: 988

I would suggest you do it with a recursive function.

In the following example you can use loadFilesAndCalculateSum() like the d3.text() but instead of one string it takes an array of strings and the callback gets the calculated sum instead of the text of a file.

/**
 * load all files and calculate the sum of the values
 * @param  {Array}    filenames Array with filenames as string
 * @param  {Function} cb        Callback function, gets the sum as param
 * @param  {number}   sum       The initial sum   
 * @param  {number}   i         Starting index for filenames
 * @return {void}
 */
function loadFilesAndCalculateSum(filenames, cb, sum, i) {
  sum = sum || 0;
  i = i || 0;
  d3.text(filenames[i], function(error, text) {
    //parse the rows and reduce them
    sum += d3.csv.parseRows(text).reduce(function(prev, curr) {
      //return previous sum + this rows sum
      return prev + d3.sum(curr, function(d){return +d;})
    },0); 

    if(i < filenames.length - 1){
      //load next file
      loadFilesAndCalculateSum(filenames, cb, sum, i+1);
    } else {
      //call the callback with the final sum
      cb(sum);
    }
  });
}

var filenames = ["file1.txt", "file2.txt", "file3.txt"];

loadFilesAndCalculateSum(filenames, function(sum){
  //do something with the total sum
  console.log(sum);
});

To clarify this. you have to do the processing of sum inside of the callback function where I put the comment do something with the total sum. This function is still executing async. That means, that everything you write after the loadFilesAndCalculateSum() function will possibly execute before the code inside the callback. You can find a little longer introduction to async javascript here

//this is executed first
//....
loadFilesAndCalculateSum(filenames, function(sum){
  //do something with the total sum
  //this is executed third, when all files are loaded and the sum is calculated
  console.log(sum);
});

//code after this point is executed second, while the files are being loaded.

If you already have a function that does something with the sum, you can pass this function to the loadFilesAndCalculateSum as second parameter. This is possible because functions are so called first class citizens:

var addthis = 5;

function doSomethingWithTheSum(sum) {
  //everything you want to do with the sum goes inside this function.

  //from here you could call other functions and pass the sum.
  soSomethingDifferentWithTheSum(sum);

  //or you use the sum inside this function
  console.log(sum);
  var newsum = sum + addthis;
  console.log(sum);

  d3.select("whatever")
      .data([sum])
      .enter()
    .append("text")
      .text(function(d){ return d;});
}

loadFilesAndCalculateSum(filenames, doSomethingWithTheSum);

You can pass functions just like any other variable. in the first example I called the second parameter of the loadFiles... function cb which is the usual abbreviation for callback. As stated in the doc comment, this param should be of type Function.

In the end of the loadFiles... function I the callback function gets called by 

....
//call the callback with the final sum
cb(sum);
....

Here the sum is given to the callback function as first parameter. Therefore if you pass a function, it should take a single param, like the anonymous function in the first example or the doSomethingWithTheSum function in the example above.

Upvotes: 1

Related Questions