CODEWITHSUNDEEP
talendtalend-mdm
Deepan Ram
Deepan Ram

Reputation: 850

Common Logging in Talend

I was trying to implement logging in Talend. So I made a job using normal components, I have recorded the error, info and debug through Twarn and Tdie. Using Logcatcher , I am segregating the log into two files of debug and error. This part is working fine.

Now I have made two jobs,

First:- Using trowgenerator generating lines, then sending to tmap and from tmap, I am sending to two twarn components based on some condition.

Second:- A job which has tlogcatcher, t_filterrow and segregating to two files based on some condition.

So I am running the second job in parallel with the first job to capture the logs, but it is not working.

I have tried several options and stuck at this point.

Main idea is to have a common logging component which can be plugged to any job for the logs.

Is there any ways with which it can be achieved ?

Upvotes: 3

Views: 6559

Answers (3)

Deepan Ram
Deepan Ram

Reputation: 850

Thanks all for the extended support. I was able to achieve what i was trying to achieve.

I have created a standard job, which is giving twarn and tdie logs. Then I have a joblet which catches the logs and distributes the logs based on error or info/debug.

Upvotes: 0

E LaRoche
E LaRoche

Reputation: 1154

Talend has built-in logging available and its very easy to setup. Step 1: create AMC database. Step 2. Set project level properties for logging. Step 3. check AMC database for logs. All errors and tWarns will be logged. Optional Step 4: create reports based on logging data in the AMC database.

You can log to file instead of db, but logging to db allows more robust query and reporting. Also if you run jobs out of the TAC you can join AMC to the TAC db for some really useful reports on your jobs.

You can log:

  • Errors and Warnings
  • Stats (job duration)
  • FlowMeter (data flow metrics)

Screen 1 shows project setup in Talend Studio

enter image description here

Screen 2 shows a query from the AMC db, log table. notice there are two other tables, one for stats, and another for flow. also notice in the row selected there is message from a tWarn.

enter image description here

Upvotes: 1

Balazs Gunics
Balazs Gunics

Reputation: 2077

Here are the 3 ways that popped in:

log4j: Enterprise version has log4j built in, which means you can set a debug level in tWarn/tDie and configure log4j to put these into different files. (When using it I found it best to have the warnings as my own info level log, since talends info level is too deep.) I'm sure log4j can be an "hacked" into open studio as well. That would solve your problem. https://exchange.talend.com has many results when searching for log4j, you should take a look.

joblet: If you prefer your current way I see 2 options, for you. First is to try and do create a joblet, however using the same file from multiple jobs is not a good idea and probably results in errors. This is not a nice solution but could work if we only talk about 1 job that needs to produce 2 files.

Project level logging: You can try to enable project level logging but that logs everything into files. It basically means redirect the logcatcher output to a file. Thats 1 file not 2, but at least it works. https://help.talend.com/display/TalendOpenStudioforBigDataUserGuide520EN/2.6+Customizing+project+settings (Stats & Logs)

There really is no nice solution to what you need.

Upvotes: 1

Related Questions