About Me

My photo
Rohit is an investor, startup advisor and an Application Modernization Scale Specialist working at Google.

Tuesday, June 14, 2016

Multi-line Java stack traces out of order in Logstash and Splunk

Do you have customers frustrated by getting multi-line Java stack traces out of order? We're working on a clean solution in our enhanced metrics work, but here is a workaround courtesy @DavidLaing .
With the Java Logback library you can do this by adding 

"%replace(%xException){'\n','\u2028'}%nopex" 

to your logging config [1] , and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will display as a new line.


mutate {
  gsub => [ "[@message]", '\u2028', "
"]
^^^ Seems that passing a string with an actual newline in it is the only way to make gsub work
}



In Splunk you will need to configure the the event stream to recombine multine exceptions into one event. See http://docs.splunk.com/Documentation/Splunk/5.0/Data/Indexmulti-lineevents

1 comment:

  1. [2] has changed to https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry/blob/develop/src/logsearch-config/src/logstash-filters/snippets/app.conf#L70-L71

    ReplyDelete