CODEWITHSUNDEEP
c#transform
callisto
callisto

Reputation: 5083

Creating an ETL system (Data import and transformation)

I have been tasked to write a module for importing data into a client's system. I thought to break the process into 4 parts:

1. Connect to the data source (SQL, Excel, Access, CSV, ActiveDirectory, Sharepoint and Oracle) - DONE
2. Get the available tables/data groups from the source - DONE
    i. Get the available fields form the selected table/data group - DONE
    ii. Get all data from the selected fields - DONE
3. Transform data to the user's requirements
4. Write the transformed data the the MSSQL target

I am trying to plan how to handle complex data transformations like:
Get column A from Table tblA, inner joined to column FA from table tblB, and concatenate these two with a semicolon in between.

OR

Get column C from table tblC on source where column tblC.D is not in table tblG column G on target database.

My worry is not the visual, but the representation in code of this operation.

I am NOT asking for sample code, but rather for some creative ideas.

The data transformation will not be with free text, but drag and drop objects that represent actions.

I am a bit lost, and need some fresh input.

Upvotes: 1

Views: 1256

Answers (2)

callisto
callisto

Reputation: 5083

See my answer: Manipulate values in a datatable?

Upvotes: 0

Felice Pollano
Felice Pollano

Reputation: 33252

maybe you can grab some ideas from this open source project: Rhino ETL.

Upvotes: 3

Related Questions