CODEWITHSUNDEEP
parsinglogginglogparser
Sushant
Sushant

Reputation: 1074

Log Parser: Parse a folder full of log files continuously

Problem

Let's suppose there is a folder A which contains log files from an application. The logging logic creates a new log file to write to every hour. I keep data for let's say 3 months. So, there maybe 10k log files in that folder at any time. I have the query to parse the files using Log parser 2.2 but the problem is that I need to export the parsed data continously to SQL.

For example, initially I parse everything that exists at that moment and export it to SQL.

From the next time onwards, I only want to parse what is new and hasn't been parsed before. New stuff maybe in a new file (in which case I can just parse the new file) but it may also be the case that some new entries have been appended to one of the files I have already parsed.

Is there a trivial way to parse only the new information in the latter?

Alternatively

Is there a way for LogParser to only parse what is new based on timestamp information when there is more than one file involved? I can put a where clause in the query which filters based on time stamp, but is it smart enough to not look at data older than the given timestamp?

Upvotes: 2

Views: 1639

Answers (1)

Sushant
Sushant

Reputation: 1074

Looks like I did not look hard enough.

The answer to continous/real time parsing with log parser is the "-icheckpoint" switch. It saves the information about how much has been parsed before so log parser does not have to repeat it when the same query is executed again.

Look at this link for a more clear example

https://www.simple-talk.com/blogs/2010/06/28/using-logparser-part-4/

Upvotes: 2

Related Questions