Splunk - Troubleshooting Skills

Table of Contents

• Tips
• Search syntax
• rex
• Show only Certain Fields
• (Not) contains a field
• See all values in a field
• Examples
• Interesting Fields
• Events before or after X seconds
• Visualizations Tab
• Write Splunk Friendly Log
• Related Posts

Tips

• Expand messages to show all fields
  • Click Format on the top and select All lines for the Max Lines setting

Search syntax

• key=value AND OR NOT
• sort num(ip) desc, -str(url) | sort -value
• reverse
• top(or rare) limit=5 name
• head 5
• highlight fa, fb
• dedup fa
• stats (stats-function(field) [AS field])… [BY field-list]
• chart count by method url
• timechart span=15m count by url
• earliest=“11/5/2017:20:00:00” latest=“11/12/2017:20:00:00”

rex

rex field=_raw "From: <(?<from>.*)> To: <(?<to>.*)>" | dedup from to | table from to
rex field=message "pause of approximately (?<time>.*)" | dedup time | table time
rex field=message "pause of approximately (?<time>.*)" | stats count by time
rex field=message ".* 0 (?<time>\d*)" | timechart max(time)

Show only Certain Fields

table field

(Not) contains a field

• NOT fa=*

See all values in a field

• top a_number SourceName

Examples

index=the_app lvl=ERROR | top limit=100 ttl, app | sort count desc
eval cluster=app+"-"+partition | top 100 cluster, ttl

Interesting Fields

• Explore and know what values for a field

Events before or after X seconds

• Show logs around current event

Visualizations Tab

Write Splunk Friendly Log

• Separate the main message and key/value pairs
• The main message should be variable-free
• Use util class (or) to help implement this

---