CODEWITHSUNDEEP
etltalend
pietà
pietà

Reputation: 770

hand coded ETL vs talend open studio

I have developed an ETL with shell scripting .

After that,I've found that there's an existing solution Talend open studio. I'm thinking of using it in my future tasks.

But my problem is that the files that i want to integrate into the database must be transformed in structure . this is the structure that i have :

19-08-02  Name                         appel    ok    hope    local  merge   (mk)   
                                                       juin    nov    sept    oct
00:00:t1  T1                            299       0      24      8      3     64
          F2                            119       0      11      8      3     62
          I1                             25       0       2      9      4     64
          F3                            105       0      10      7      3     61
          Regulated F2                    0       0       0
          FR T1                         104       0      10      7      3     61

i must transform it into a flat file format .

Do talend offer me the possibility to do several transformations before integrating data from csvfiles into the databaseor not ?

Edit

this is an example of the flat file that i want to acheive before integrating data to the database (only first row is concerned) :

Timer,T1,F2,I1,F3,Regulated F2,FR T1
00:00:t1,299,119,25,105,0,104  
00:00:t2,649,119,225,165,5,102
00:00:t5,800,111,250,105,0,100

Upvotes: 1

Views: 283

Answers (1)

tobi6
tobi6

Reputation: 8239

We can split the task into three pieces, extract, transform, load.

Extract

First you have to find out how to connect to the source. With Talend its possible to connect to different kinds of sources, like databases, XML files, flat files, csv etc. pp. They are called tFileInput or tMySQLInput to name a few.

Transform

Then you have to tell Talend how to split the data into columns. In your example, this could be the white spaces, although the splitting might be difficult because the field Name is also split by a white space.

Afterwards, since it is a column to row transposition, you have to write some Java code in a tJavaRow component or could alternatively use a tMap component with conditional mapping: (row.Name.equals("T1") ? row.value : 0)

Load

Then the transformation would be completed and your data could be stored in a database, target file, etc. Components here would be called tFileOutput or tOracleOutput for example.

Conclusion

Yes, it would be possible to build your ETL process in Talend. The transposition could be a little bit complicated if you are new to Talend. But if you keep in mind that Talend processes data row by row (as your script does, I assume) this is not that big of a problem.

Upvotes: 2

Related Questions