D3 - Loading data from a second CSV

Question

I have two separate CSV files, sharing an attribute:

File 1

name,total
Frank,13
Mary,24
Jim,46

File 2

name,desc
Frank,yellow
Mary,blue
Jim,green

How can I map the attribute desc to the File 1 so, let's say, when I hover on Frank, I will see "13" and "yellow"?

Currently, my code is looking like this:

d3.csv("data/csv/bilancio-missioni-desc.csv", function(descrizione) {
  data = descrizione.map(function(data){
            div.html("<h3>" + d.name + "</h3>" + "<p>" + data.desc + "</p>")
})

The problem is that d.name and data.desc (from File 1 and File 2) don't match — I can understand it's because I didn't combine them so they can share the common attribute name, but I don't know how to write the right piece of code.

UPDATE My code so far:

d3.csv("data/csv/bilancio-tipologiedispesa-nest.csv", function(data1) {
d3.csv("data/csv/bilancio-missioni-desc.csv", function(data2) {
    //code that depends on both data1 and data2       
    data1.forEach(d => {
        data2.forEach(e => {
        if (d.name === e.name) {
        d.desc = e.desc;
    }
});
}); 
// Fade all the segments.
d3.selectAll("path")
.style("opacity", .3);
vis.selectAll("path")
.filter(function(node) {
    return (sequenceArray.indexOf(node) >= 0);
})
.style("opacity", 1);
div.transition()
.duration(200)
.style("opacity", .9);
div.html("<h3>" + d.name + "</h3>" + "<p>" + d.desc + "</p>");
});
});
}

If I console.log(data1), "desc" (from data2) is appended to data1 (yay!). But "d.desc" is returning "unidefined" in the HTML.

Answer 1

As Gerardo Furtado has already laid out in his answer you will have to resort to either using d3.queue or a nested version of d3.csv to keep loading your data in sync. When making use of two built-in features of D3 the nested approach can be refined to free you from doing explicit nested looping after the initial loading.

• Use a D3 Map to provide for fast access to your data by the name property.

• Use a row conversion function which may be passed as the second, optional parameter to d3.csv(url[[, row], callback]):

  If a row conversion function is specified, the specified function is invoked for each row, being passed an object representing the current row (d)…

  This comes in particularly handy as the rows will internally be iterated over while parsing the file's content. You can use the row conversion function as a hook to be able to sync the second file's data to the first file's data by manipulating the map within that function.

I set up a Block demonstrating this approach:

d3.csv("file1.csv", function(data1) {
  // 1. Build a D3 map from your first file's contents 
  var map = d3.map(data1, function(d) { return d.name; });

  d3.csv("file2.csv", function(row) {
    // 2. Use the row function of the second file to match your data.
    map.get(row.name).desc = row.desc;  // 3. Sync data
    return null;   // 4. Don't append any row to the array.
  }, function() {  // 5. There is no parameter present; all data has been synced into data1.
    // Just logging below
    console.log(data1);  // [{ name: "Frank", total: "13", desc: "yellow"}, ...]
  })
});

Let me talk you through this step by step:

1. We build a map based on the parsed data from your first file, using the name property as the key.

2. When doing the inner loading of the second file we specify a row conversion as the second parameter. Normally, this will be used to do some data conversion and the like and return some converted object representing the actual row's data. In our scenario, however, we are not interested in the conversion itself, but in the implicit iteration and the access to the row's data.

3. We are now ready to sync the data by getting the object corresponding to this row's name from the map.

4. Returning null from the row conversion function will avoid building up an array for the second file, since we are not interested in a standalone version of the content:

   If the returned value is null or undefined, the row is skipped and will be ommitted from the array…

5. Within the callback of the inner d3.csv() your data is ready and synced. Notice, that we are not passing any parameter to this callback, because we will access the synced data via data1. We will not even have to bother about the map anymore, which was but a means to allow fast access to our data from the first file. Because the map stores references to our data objects, we updated the actual objects from data1, which were used to build the map.

---

As a refinement, you might want to add some ES6 sugar making this even more concise (ES6 Block):

d3.csv("file1.csv", data1 => {
  let map = d3.map(data1, d => d.name);
  d3.csv("file2.csv",
         row => (map.get(row.name).desc = row.desc, null),
         () => {console.log(data1)}  // Merged data available at this point.
  );
}); 

Answer 2

There are two main approaches for loading several CSVs: using d3.queue() or nesting them:

d3.csv("file1.csv", function(data1) {
    d3.csv("file2.csv", function(data2) {
        //code that depends on both data1 and data2
    })
});

Where data1 is the array from "file1.csv" and data2 is the array from "file2.csv".

Then, we're gonna merge the two arrays based on their properties.

You can do this creating a third array or pushing the values in one of the two original arrays. Here, I'll do the second option (actually, the second option is what you asked: "How can I "map" the attribute 'desc' to the File 1").

Have in mind that this is an ad hoc solution I made specifically to your problem (it will not work for another data structure):

data1.forEach(d => {
    data2.forEach(e => {
        if (d.name === e.name) {
            d.desc = e.desc
        }
    })
})

Now, data1 has all the information you want: name, total and desc.

Here is a demo, check the console with the data (In this demo, I'm using a <pre> element to load the data, because I cannot use a CSV in the snippet):

var data1 = d3.csvParse(d3.select("#file1").text());
var data2 = d3.csvParse(d3.select("#file2").text());

data1.forEach(d => {
    data2.forEach(e => {
        if (d.name === e.name) {
            d.desc = e.desc
        }
    })
})

console.log(data1);

pre {
    display: none;
}

<script src="https://d3js.org/d3.v4.min.js"></script>
<pre id="file1">name,total
Frank,13
Jim,46
Mary,24</pre>
<pre id="file2">name,desc
Frank,yellow
Mary,blue
Jim,green</pre>

PS: If you files are too big, have a look at @altocumulus alternative code in the comments below.

Answer 3

As mentioned in another answer you can nest the csv loading function to to load the files asynchronously. Then you can combine them together:

var file1 = "data/csv/bilancio-missioni-desc.csv";
var file2 = "data/csv/bilancio-missioni-info.csv"; // I named it info just as an example

var div = d3.select('#my-div');
var data = {};

d3.csv(file1, function(descrizione) {
    d3.csv(file2, function(info) {
        processData(descrizione, info);
    })
})

function processData(descrizione, info) {
    descrizione.forEach(d => data[d.name] = d);
    info.forEach(i => data[i.name] && data[i.name]['desc'] = i.desc);
    appendData();
}

function appendData() {
    div.data(data).append(d => '<h3>' + d.name + ' (' + d.total ')</h3> <p>' + d.desc + '</p>');
}

This code might not work without a few adjustments of course, but I hope you get the idea. If not, let me know if there's something unclear to you. Good luck!