Tools - Troubleshooting Skills

Splunk

Search syntax

• key=value AND OR NOT
• | sort num(ip) desc, -str(url) | sort -value
• | reverse
• | top(or rare) limit=5 name
• | head 5
• | highlight fa, fb
• | dedup fa
• | stats count by method
• | chart count by method url
• | timechart span=15m count by url

earliest=“11/5/2017:20:00:00” latest=“11/12/2017:20:00:00”

Show only Certain Fields

| table field

(Not) contains a field

NOT fa=*

See all values in a field

| top a_number SourceName

Interesting Fields

• Explore and know what values for a field

Events before or after X seconds

• Show logs around current event

Visualizations Tab

Linux

crash or reboot?

ast -n2 -x shutdown reboot

• If this reports a SYSTEM_SHUTDOWN followed by a SYSTEM_BOOT, all is well; however, if it reports 2 SYSTEM_BOOT lines in a row, then clearly the system did not shutdown gracefully

host $ip

Write Troubleshooting Friendly Program

• Name your thread

Tips, Tricks and Gotchas

Chrome
Remove pdf password
Chrome -> Print -> Save As PDF
chrome://settings/cookies
chrome://settings/content/notifications?search=Notifications

Chrome Dev Tool
Bring Default Context Menu: Enable Copy Text
document.designMode = "on";

Command Menu - Cmd + Shift + P
Capture screenshots without extensions

Blogger
You have logged out from another location
Common approaches to handle website related issues: clear cache and cookie

Search for multiple labels
search/label/LABEL1+LABEL2
/search/?q=label:LABEL1|label:LABEL2

Gmail
Sometimes Gmail doesn't sort the replies by date in one conversation - maybe when different people reply different thread.

But Gmail seems already display threads that you have not read

- Example: Robinhood pdf

Robinhood upcoming earnings
https://robinhood.com/applink/upcoming_earnings

Lessons Learned about Programming and Soft Skills - 2018

Make things easier to (re)use
Example: LocalConnectToIF1, make other devs easier to use this trick by just setting a property, all other steps are coded.

Make it easier to search later

Add tag/text to help search: to-read, to-post etc
Bookmark or save important/useful links
Add it to favorites

Assume other person is probably doing the right thing
- You think it should implement like this, check the code/design, maybe it's already done this way, or maybe they have good reason to choose a different approach

Do research first
- usually the same problem has already been solved

Do some (quick, simple) (re)search on unknown things
- try to understand it, solve your questions
- is it good, when to use it, who use it, why?

- Example: bitcoin, block chain

Make applications work similar/same across all platforms
- unless there is good reason to not
Example: Robinhood Android vs iPhone

Ask questions before take time-consuming operation
Example: crossroads

Effective SearchReframe the search, be more specific

Example: chat scalable design -> websocket load balancing

Lessons Learned about Programming and Soft Skills - 2017

Programmers' Toolbox

Kafka
(different levels) Retry Queue and Dead Letter Queen

Redis
SETNX key value
LRU, LRU
- Sample 3 or 5 keys, maintain a larger pool of candidate keys (16)
Pipeline
- Queue in client side
Translation
- Multi, exec, abort
- queued in server side
- Continue to execute even some commands fail, no rollback
- Use Watch to implement CAS, fail the translation if the key changes
- pub/sub
- sorted sort(zset), 
- hyperloglog, PFADD, PFCOUNT

- service side lua script

OkHttp

WebSocket
- efficiency: persistent duplex, bidirection connection

Java - @ServerEndpoint
Socket.io

LevelDB/RocksDB

jq
jq '.pods[] | .name, .replication'
'.owner.name'
'.likes[0]'
.[] return all elements in the array