Join two csv files (1-N relation)

Question

The csv files are tab delimited

file1.csv:

id_album   name        date
001        Nevermind   24/09/1991
...

file2.csv:

id_song   id_album   name  
001       001        Smells Like Teen Spirit
002       001        In Bloom
...

I would like to obtain this output.csv :

id_album   name        date         songs
001        Nevermind   24/09/1991   001,Smells Like Teen Spirit,002,In Bloom,...

Do you see a way to do it in Bash (preferably) or Python ?

I have a lot of records in my csv files (millions of lines).

EDIT

I tried join / sed / awk but was not able to manage the 1 to N relation

Answer 1

TL;DR

from io import StringIO
file1 = """id_album,name,date
001,Nevermind,24/09/1991"""

file2 = """id_song,id_album,name
001,001,Smells Like Teen Spirit
002,001,In Bloom"""

df1 = pd.read_csv(StringIO(file1))
df1 = df1.rename(columns={'name':'album_name'})

df2 = pd.read_csv(StringIO(file2))
df2 = df2.rename(columns={'name':'song_name'})


df3 = df1.merge(df2, on='id_album')
df4 = pd.DataFrame(list({album['id_album'].unique()[0]:','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])}.items()), columns=['id_album', 'song_id_name'])

df_want = df1.merge(df4)

[out]:

>>> df_want
   id_album album_name        date                          song_id_name
0         1  Nevermind  24/09/1991  1,Smells Like Teen Spirit,2,In Bloom

---

In Long

Given:

>>> from io import StringIO
>>> file1 = """id_album,name,date
... 001,Nevermind,24/09/1991"""

>>> file2 = """id_song,id_album,name
... 001,001,Smells Like Teen Spirit
... 002,001,In Bloom"""

>>> df1 = pd.read_csv(StringIO(file1))
>>> df1 = df1.rename(columns={'name':'album_name'})

>>> df2 = pd.read_csv(StringIO(file2))
>>> df2 = df2.rename(columns={'name':'song_name'})

>>> df1
   id_album album_name        date
0         1  Nevermind  24/09/1991

>>> df2
   id_song  id_album                   name  
0        1         1  Smells Like Teen Spirit
1        2         1                 In Bloom

First merge the 2 DataFrames on id_album column:

>>> df3 = df1.merge(df2, on='id_album')
>>> df3
   id_album album_name        date  id_song                song_name
0         1  Nevermind  24/09/1991        1  Smells Like Teen Spirit
1         1  Nevermind  24/09/1991        2                 In Bloom

Now for some pandas trick:

1. First group the rows by the `id_album` column:
2. In each group, get the `id_song` and `song_name` columns and stack them

>> [','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])]
['1,Smells Like Teen Spirit,2,In Bloom']

In a similar manner, get the album_name by from .groupby():

>>> [album['album_name'].unique()[0] for idx, album in df3.groupby(['id_album'])]
['Nevermind']

Lets combine the two groupby operations:

>>> {album['album_name'].unique()[0]:','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])}
{'Nevermind': '1,Smells Like Teen Spirit,2,In Bloom'}

>>> album2songs = {album['album_name'].unique()[0]:','.join(list(album[['id_song', 'song_name']].astype(str).stack())) for idx, album in df3.groupby(['id_album'])}

Put that album2songs into a dataframe:

>>> df4 = pd.DataFrame(list(album2songs.items()), columns=['album_name', 'song_id_name'])
>>> df4
  album_name                          song_id_name
0  Nevermind  1,Smells Like Teen Spirit,2,In Bloom

Now join df1 and df4:

>>> df1.merge(df4)
   id_album album_name        date                          song_id_name
0         1  Nevermind  24/09/1991  1,Smells Like Teen Spirit,2,In Bloom

BTW, @RomanPerekhrest awk solution is way cooler!

Answer 2

Discover awk language:

awk -F'[[:space:]][[:space:]]+' 'NR==FNR{ if(NR>1) a[$2]=($2 in a? a[$2]",":"")$1","$3; next}
       FNR==1{ print $0,"songs" }
       $1 in a{ print $0,a[$1] }' file2.csv OFS='\t' file1.csv > output.csv

The output.csv content:

id_album   name        date songs
001        Nevermind   24/09/1991   001,Smells Like Teen Spirit,002,In Bloom