CODEWITHSUNDEEP
hadooploggingelasticsearchlog-analysistransactionloganalysis
AngryJS
AngryJS

Reputation: 955

Log analysis using hadoop to find error scenarios and frequent repeatative transactions

I have a business scenario wherein I want have my application server logs and I want to accomplish below tasks -

  1. Find out the error scenarios captured in log to find error code [in log], exception caused, time.
  2. repeatative transactions done by some user possibly a security breach.
  3. alert admin when above two happens.

I can do that using some multithreaded basic java application having batch processing but I want to do that using big data technologies and plus want to have some graphical representation of the application behaviour as well.

Can someone suggest me the best tech to be used in order to achieve this pretty fast and scalable? and if possible architecture of how to do that?

It will be great if someone cal help with that

Upvotes: 0

Views: 225

Answers (1)

uk2016
uk2016

Reputation: 11

Have you thought about implementing a big data solution? I have a similar requirement. My data source is weblogic domain application log. Here is my approach. 1. Stream the weblogic application error log into hadoop using flume or other streaming tools. 2. Load the data into Spark Dataframe. 3. Write Spark SQL queries to analyze the error data

We have a DB error log table. I will use that as another data source to correlate the Web logic DB exceptions. The weblogic error data is CSV format, separated by two pipe symbols ("||"). But, the problem with the input data is, the last column data is spread over to multiple lines as shown below. Spark treats the continuation of last column on next line as "new line" and hence the load is getting failed. Appreciate if anyone has any thought on how to deal this issue.

||20160704||01:58:32,294||396c0a8e2470e7a21467611910768||com.seic.dataservices.impl.InstrumentSearchDoImpl||[ACTIVE] ExecuteThread: '9' for queue: 0) ....INSTRUMENT_ID(1004915) PRICE_DATE(01-JUL-16) does not exist in Table TABEL_NAME. Price data not found.. ORA-06512: at "Qxx_xxx.ERROR_PKG", line 502 ORA-06512: at "Qxx_xxx.IM_PRICING", line 6221 ORA-06512: at line 1 )

Upvotes: 0

Related Questions