About Me

My photo
Rohit is an investor, startup advisor and an Application Modernization Scale Specialist working at Google.

Tuesday, June 14, 2016

Multi-line Java stack traces out of order in Logstash and Splunk

Do you have customers frustrated by getting multi-line Java stack traces out of order? We're working on a clean solution in our enhanced metrics work, but here is a workaround courtesy @DavidLaing .
With the Java Logback library you can do this by adding 


to your logging config [1] , and then use the following logstash conf.[2]
Replace the unicode newline character \u2028 with \n, which Kibana will display as a new line.

mutate {
  gsub => [ "[@message]", '\u2028', "
^^^ Seems that passing a string with an actual newline in it is the only way to make gsub work

In Splunk you will need to configure the the event stream to recombine multine exceptions into one event. See http://docs.splunk.com/Documentation/Splunk/5.0/Data/Indexmulti-lineevents

1 comment:

  1. [2] has changed to https://github.com/cloudfoundry-community/logsearch-for-cloudfoundry/blob/develop/src/logsearch-config/src/logstash-filters/snippets/app.conf#L70-L71


Note: Only a member of this blog may post a comment.