How to make the logstash 2.3.2 configuration file more flexible

Question

I am using logstash 2.3.2 to read and parse log file of wso2 esb. I am able to successfully parse the log entries and send them to an API in Json format.

In Log file there are different log levels such as "INFO,ERROR, WARN and DEBUG". Currently, I am only sending a log entry through, If its logtype is Error.

Sample Log File:

TID: [-1234] [] [2016-05-26 11:22:34,366]  INFO {org.wso2.carbon.application.deployer.internal.ApplicationManager} -  Undeploying Carbon Application : CustomerService_CA_01_001_1.0.0... {org.wso2.carbon.application.deployer.internal.ApplicationManager}
TID: [-1234] [] [2016-05-26 11:22:35,539]  INFO {org.apache.axis2.transport.jms.ServiceTaskManager} -  Task manager for service : CustomerService_01_001 shutdown {org.apache.axis2.transport.jms.ServiceTaskManager}
TID: [-1234] [] [2016-05-26 11:22:35,545]  INFO {org.apache.axis2.transport.jms.JMSListener} -  Stopped listening for JMS messages to service : CustomerService_01_001 {org.apache.axis2.transport.jms.JMSListener}
TID: [-1234] [] [2016-05-26 11:22:35,549]  INFO {org.apache.synapse.core.axis2.ProxyService} -  Stopped the proxy service : CustomerService_01_001 {org.apache.synapse.core.axis2.ProxyService}
TID: [-1234] [] [2016-05-26 11:22:35,553]  INFO {org.wso2.carbon.core.deployment.DeploymentInterceptor} -  Removing Axis2 Service: CustomerService_01_001 {super-tenant} {org.wso2.carbon.core.deployment.DeploymentInterceptor}
TID: [-1234] [] [2016-05-26 11:22:35,572]  INFO {org.apache.synapse.deployers.ProxyServiceDeployer} -  ProxyService named 'CustomerService_01_001' has been undeployed {org.apache.synapse.deployers.ProxyServiceDeployer}
TID: [-1234] [] [2016-05-26 18:10:26,465]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: LogaftervalidationWSAction: urn:mediateLogaftervalidationSOAPAction: urn:mediateLogaftervalidationMessageID: urn:uuid:f89e4244-7a95-46ff-9df2-3e296009bf8bLogaftervalidationDirection: response {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-05-26 18:10:26,469]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: XPATH-LogLastNameWSAction: urn:mediateXPATH-LogLastNameSOAPAction: urn:mediateXPATH-LogLastNameMessageID: urn:uuid:f89e4244-7a95-46ff-9df2-3e296009bf8bXPATH-LogLastNameDirection: responseXPATH-LogLastNameproperty_name LastName_Value = XPATH-LogLastNameEnvelope:
TID: [-1234] [] [2016-05-26 18:10:26,477] ERROR {org.apache.synapse.mediators.transform.XSLTMediator} -  The evaluation of the XPath expression //tns1:Customer did not result in an OMNode : null {org.apache.synapse.mediators.transform.XSLTMediator}
TID: [-1234] [] [2016-05-26 18:10:26,478] ERROR {org.apache.synapse.mediators.transform.XSLTMediator} -  Unable to perform XSLT transformation using : Value {name ='null', keyValue ='gov:CustomerService/01/xslt/CustomertoCustomerSchemaMapping.xslt'} against source XPath : //tns1:Customer reason : The evaluation of the XPath expression //tns1:Customer did not result in an OMNode : null {org.apache.synapse.mediators.transform.XSLTMediator}
org.apache.synapse.SynapseException: The evaluation of the XPath expression //tns1:Customer did not result in an OMNode : null
    at org.apache.synapse.util.xpath.SourceXPathSupport.selectOMNode(SourceXPathSupport.java:100)
    at org.apache.synapse.mediators.transform.XSLTMediator.performXSLT(XSLTMediator.java:216)
    at org.apache.synapse.mediators.transform.XSLTMediator.mediate(XSLTMediator.java:196)
    at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
    at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
    at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
    at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:214)
    at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:81)
    at org.apache.synapse.mediators.AbstractListMediator.mediate(AbstractListMediator.java:48)
    at org.apache.synapse.mediators.base.SequenceMediator.mediate(SequenceMediator.java:149)
    at org.apache.synapse.core.axis2.ProxyServiceMessageReceiver.receive(ProxyServiceMessageReceiver.java:185)
    at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
    at org.apache.synapse.transport.passthru.ServerWorker.processEntityEnclosingRequest(ServerWorker.java:395)
    at org.apache.synapse.transport.passthru.ServerWorker.run(ServerWorker.java:142)
    at org.apache.axis2.transport.base.threads.NativeWorkerPool$1.run(NativeWorkerPool.java:172)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:744)
TID: [-1234] [] [2016-05-26 18:10:26,500]  INFO {org.apache.synapse.mediators.builtin.LogMediator} -  To: , WSAction: urn:mediate, SOAPAction: urn:mediate, MessageID: urn:uuid:f89e4244-7a95-46ff-9df2-3e296009bf8b, Direction: response {org.apache.synapse.mediators.builtin.LogMediator}
TID: [-1234] [] [2016-05-26 11:32:24,272]  WARN {org.wso2.carbon.core.bootup.validator.util.ValidationResultPrinter} -  The running OS : Windows 8 is not a tested Operating System for running WSO2 Carbon {org.wso2.carbon.core.bootup.validator.util.ValidationResultPrinter}
TID: [-1234] [] [2016-05-26 11:32:24,284]  WARN {org.wso2.carbon.core.bootup.validator.util.ValidationResultPrinter} -  Carbon is configured to use the default keystore (wso2carbon.jks). To maximize security when deploying to a production environment, configure a new keystore with a unique password in the production server profile. {org.wso2.carbon.core.bootup.validator.util.ValidationResultPrinter}
TID: [-1] [] [2016-05-26 11:32:24,315]  INFO {org.wso2.carbon.databridge.agent.thrift.AgentHolder} -  Agent created ! {org.wso2.carbon.databridge.agent.thrift.AgentHolder}

Configuration File:

input {
 stdin {}
    file {
       path => "C:\MyDocument\Project\SampleESBLogs\wso2carbon.log" 
        type => "wso2carbon"
        start_position => "beginning"
        codec => multiline {
                pattern => "(^\s*at .+)|^(?!TID).*$"
                negate => false
                what => "previous"
        }

    }
}
filter {

    if [type] == "wso2carbon" {
        grok {
            match => [ "message", "TID:%{SPACE}$$%{INT:log_SourceSystemId}$$%{SPACE}$$%{DATA:log_ProcessName}$$%{SPACE}$$%{TIMESTAMP_ISO8601:TimeStamp}$$%{SPACE}%{LOGLEVEL:log_MessageType}%{SPACE}{%{JAVACLASS:log_MessageTitle}}%{SPACE}-%{SPACE}%{GREEDYDATA:log_Message}" ]
            add_tag => [ "grokked" ]        
        }

        if "grokked" in [tags] {
            grok {
                match => ["log_MessageType", "ERROR"]
                add_tag => [ "loglevelerror" ]
            }   
        }

        if !( "_grokparsefailure" in [tags] ) {
            grok{
                    match => [ "message", "%{GREEDYDATA:log_StackTrace}" ]
                    add_tag => [ "grokked" ]    
                }
            date {
                    match => [ "timestamp", "yyyy MMM dd HH:mm:ss:SSS" ]
                    target => "TimeStamp"
                    timezone => "UTC"
                }
        }               
    }
}

    if ( "multiline" in [tags] ) {
        grok {
            match => [ "message", "%{GREEDYDATA:log_StackTrace}" ]
            add_tag => [ "multiline" ]
            tag_on_failure => [ "multiline" ]       
        }
        date {
                match => [ "timestamp", "yyyy MMM dd HH:mm:ss:SSS" ]
                target => "TimeStamp"

        }       
    }

}

output {
       if [type] == "wso2carbon" {  
        if "loglevelerror" in [tags] {
            stdout { }
            http {
                url => "https://localhost:8086/messages"
                http_method => "post"
                format => "json"
                mapping => ["TimeStamp","%{TimeStamp}","MessageType","%{log_MessageType}","MessageTitle","%{log_MessageTitle}","Message","%{log_Message}","SourceSystemId","%{log_SourceSystemId}","StackTrace","%{log_StackTrace}"]
            }
        }
    }
}

Problem Statement :

I want to provide a flexible option to user that from where user can decide that which type of log entries need to be send towards API?. Like in the existing setup only "ERROR" type log entries are being sent towards the API.

Currently How I am doing this :

Currently I am doing this in the following way. I am first checking in my filter that If a recenlty parsed log entry has error type, then add a tag to that log entry.

if "grokked" in [tags] {
            grok {
                match => ["log_MessageType", "ERROR"]
                add_tag => [ "loglevelerror" ]
            }   
        }

And in the output section, I am againg checking in an "if" condition that if the parsed entry has the required tag, then let it go otherwise drop it or ignore it.

if "loglevelerror" in [tags] {
            stdout { }
            http {
             ....
          }
      }

Now, I also want to check for other log levels as well, So is there any other better way of doing this?. Or I have to put in place similar If blocks with same sttuff inside them just the condition will be different.

To Sum it up: If I want to provide somebody the option that by using my configuration they can choose via uncommenting or by any other way, which type of log entry (INFO,WARN,ERROR,DEBUG) they want to send to API, How Can I acheieve that?

How to make the logstash 2.3.2 configuration file more flexible

Answers (1)

Related Questions