CODEWITHSUNDEEP
mysqlsql-serverdatabasetalend
Sunil Rukhaya
Sunil Rukhaya

Reputation: 27

How to Implement logging at the end of each job In talend?

I am new to Talend os.

However, I received a task:

  1. Create file delimited .csv metadata (one for Lead & Opportunity).
  2. Move files to your repository on the AWS server (the etl_process1 login).
  3. Create two tables sfdc_leads_reporting_raw and sfdc_opp_reporting_raw.
  4. Load the data from the files into the tables. Ensure the data types are correctly used when creating metadata schemas & tables.

Till step 4 I am done.

Now the problem is:

How to Implement logging at the end of each job to report the number of leads (count of distinct id in leads table) and number of opportunities created (count of opportunity id) by stages (how many converted, qualified, closed won, and dead)?

Help would be appreciated.

Upvotes: 1

Views: 245

Answers (1)

Ibrahim Mezouar
Ibrahim Mezouar

Reputation: 4051

You can get this data using global variables, in a subjob at the end of your job. Most components provide a global variable called tComponent_NB_LINE (or _NB_LINE_INSERTED for database components) that gives you the number of lines output by the component. For instance tFileOutputDelimited_1_NB_LINE or tOracleOutput_1_NB_LINE_INSERTED.

Using these variables you can log into console or file. Here is a simple example. If you have a tOracleOutput_1 in your job you can do:

tPostJob -- OnComponentOk -- tFixedFlowInput -- Main -- tLogRow

Inside tFixedFlowInput you retrieve the variable

(Integer)globalMap.get("tOracleOutput_1_NB_LINE_INSERTED")`.

If you need to log aggregated info, you can append a tAggregateRow to your output components, and use tSetGlobalVar to get count by certain criteria.

Upvotes: 1

Related Questions