CODEWITHSUNDEEP
elasticsearchlogstashkibanaelastic-stackfilebeat
ZSH
ZSH

Reputation: 661

Filebeat multiline filter doesn't work with txt file

I use the filebeat to collect data from .txt file.I'm trying to use Filebeat multiline capabilities to combine log lines into one entry using the following Filebeat configuration:

filebeat.inputs:

- type: filestream
  enabled: true
  multiline.pattern: '^[0-9]{2}\/[0-9]{2}\/[0-9]{4}'
  multiline.negate: true
  multiline.match: after
  paths:
    - .\My.log

output.logstash:
  hosts: ["localhost:5044"]

This is an example of logs. I want to merge stack trace logs.

18/11/2021 19:17:25,717 [96] ERROR B2XPPA.Web.UI.Utilities.GlobalExceptionFilter -
System.AggregateException: One or more errors occurred. ---> System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceledExceptions)
   at System.Threading.Tasks.Task`1.GetResultCore(Boolean waitCompletionNotification)
   at System.Threading.Tasks.Task`1.get_Result()
   at B2X.AddressRestfulAPIClient.RestfulWebAPIClient.GetDetails(String url)
   at B2X.AddressValidation.AddressValidation.GetEircodeAddress(String Eircodeid)
   at B2XPPA.Web.UI.Services.DropdownsPPAService.GetEircodeAddress(String eirCodeId)
   at B2XPPA.Web.UI.Models.Populators.PPADetailsPopulator.Populate(IViewModel viewModel, Quote quote)
   at B2X.Services.ApplicationService.Application`1.PopulateDetailsView(DetailsViewModel viewModel, String referenceNumberInController, Quote quote)
   at B2XPPA.Web.UI.Controllers.QuoteController.Create(QuoteDetailsViewModel viewModel)
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters)
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)

However, all the logs in Kibana continue to be separated at each new line, and no multiline formatting is happening at all. Can anyone help me out with this?

Upvotes: 1

Views: 5431

Answers (2)

ZSH
ZSH

Reputation: 661

I've found out what was the problem. I used filestream as a filebeat input. multiline doesn't work with filestream. I've changed the type to log then everything works fine.

filebeat.inputs:
 multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
 multiline.negate: true
 multiline.match: after
- type: log
  enabled: true

  paths:
    - .\My.log

output.logstash:
  hosts: ["localhost:5044"]

Upvotes: 5

leandrojmp
leandrojmp

Reputation: 7463

Your multiline pattern is not matching anything.

The pattern ^[0-9]{4}-[0-9]{2}-[0-9]{2} expects that your line to start with dddd-dd-dd, where d is a digit between 0 and 9, this is normally used when your date is something like 2022-01-22

But your line starts with the following pattern dd/dd/dddd, so you would need to change your multiline pattern to match the start of your lines.

This pattern ^[0-9]{2}\/[0-9]{2}\/[0-9]{4} would match lines starting with dates like the one you have, for example 18/11/2021.

Upvotes: 0

Related Questions