Essential Linux Commands for Developers

Shortcuts:
Ctrl + a - Move to the start of line
Ctrl + e - Move to the end of line
Clear the screen:  Ctrl + l
Search as you type. Ctrl + r and type the search term; Repeat Ctrl + r to loop through results.

!$
tac: print a file line by line in reverse order.

List all open ports
netstat -ltn -- all process that listens on tcp ports
-udp|-u
--tcp|-t
--listening
--program|-p

Which process opens 9160
lsof -i :9160
lsof -p pid1
lsof /var/log/system.log
List opened files under a directory
lsof +D /var/log/
List files opened by a specific user
lsof -u user
lsof -u ^user
lsof -i

iptables
To get a complete presentation of the netfilter rules
iptables -vL -t filter
iptables -vL -t nat
iptables -vL -t mangle
iptables -vL -t raw

iptables -vL -t security

nohup someCommand > someFile.log 2>&1 &

sort | uniq -d
     -u      Only output lines that are not repeated in the input.
     -d      Only output lines that are repeated in the input.
     -c      count

find /usr -size +10M
find . -exec xxx {} \;

Find configuration data
find /etc -type f -exec grep -FHi "ENCRYPTION_PASSWORD" {} +
grep -r ENCRYPTION_PASSWORD /etc 2>/dev/null

find /etc -type f -print0  | xargs -0 grep ENCRYPTION_PASSWORD

Find biggest files
find ~/ -type f -exec du -Sh {} + | sort -rh | head -n 5

How long a command takes
time curl ""

List all functions
declare -f
declare -F - only list function names
declare -f function_name

Truncate a file
cat /dev/null > file
> file

Generate random number
/dev/random block when entropy pool is exhausted
/dev/urandom will not block

echo $RANDOM
od -An -N1 -i /dev/urandom
od -An -N2 -i /dev/urandom
for i in {1..5}; do echo $RANDOM; done

Display disk usage
du -sm
List all directories and their total size:
du -sh *
-s: Display an entry for each specified file. (Equivalent to -d 0)
Show only total for each directories

du -h -d 1

List hidden files:  ls -a | grep "^\."
List files and sorted by size: ls -l | grep ^- | sort -nr
List link files: ls -l | grep '^l'
ls -l --block-size=M
find . -type d -maxdepth 1 -name "H2*"
ls -d H2*/

Grep:
-c --count: Show count of matched lines
-E, --extended-regexp - same as egrep
-n, --line-number

grep -E '^abc(worda|wordb)' /etc/group
-n or --line-number
-A NUM, --after-context=NUM
       Print NUM lines of trailing context after matching lines.
-B NUM, --before-context=NUM
       Print NUM lines of leading context before matching lines.
-C NUM, --context=NUM
       Print NUM lines of output context. 
-w - searched as a word
-o, --only-matching
Prints only the matching part of the lines.
-l  -- only show matched file name
-w -- only if it's a whole word
-r  -- recursively search

Grep file that contains binary data
cat -v tmp/test.log | grep regex
-v      Display non-printing characters so they are visible.

Search specific line ranges:
sed -n 'startLine,endLinep' a.txt | grep XX 

Use extended regular expression with grep -E

scroll results and pagination
grep ** | more/less 

sed s/word1/word2/g fileName
Only display nth line:         sed -n 'n p' file
Delete nth line:          sed 'n d' file > newFile
Delte nth line in place:  sed –i 'n d' file.txt

Remove last line:          sed –i '$ d' file.txt
-i change in place
Delete first line: sed –i '$ d' file.txt
sed –i 'm,n d' file.txt
sed -n 'n p' file.txt | wc -c
-i extension
Edit files in-place, saving backups with the specified extension.  If a zero-length extension is given, no backup will be saved.

xargs
echo a b c | xargs echo
find /tmp -name "*.bk" -type f -print | xargs /bin/rm -f
find /tmp -name "*.bk" -print0 | xargs -0 -I {} mv {} ~/bk.files
-- better: find /tmp -depth -name "*.bk" -type f -delete
find /tmp -name "*.bk" -print0 | xargs -0 -I file mv file ~/bk.files
cut -d: -f1 < /etc/passwd | sort | xargs echo

-I replstr
--null, -0 - handle spaces in file name
Change xargs to expect NUL (``\0'') characters as separators, instead of spaces and newlines.  This is expected to be used in concert with the -print0 function in find

less +F the.log
pv - monitor the progress of data through a pipe
pv the.gz | gunzip
jq - sed for JSON
curl the_url | jq

python -m SimpleHTTPServer
| python -mjson.tool

Scripting
if [[ -f ~/.bashrc ]]; then
   source ~/.bashrc
fi
[ ! -f $FILE ] && { echo "$FILE not found"; exit -1; }

Use $( ... ), not `` to capture command output
var="$(command "$(command1)")"
$(( $a+$b )) to execute arithmetic expressions
Put ; do and ; then on the same line as the while, for or if.
Prefer brace-quoting all other variables.
Prefer "${var}" over "$var"
Use "$@" unless you have a specific reason to use $*.
- "$@" will retain arguments as-is, so no args provided will result in no args being passed on;
- "$*" expands to one argument, with all args joined by (usually) spaces, so no args provided will result in one empty string being passed on.

while IFS=, read var1 var2 var3; do
  ...

done < file.txt


Readonly
Make variable readonly: readonly var=value
Make function readonly: readonly -f function
readonly -p/-f

Use Local Variables
local var="something"
local var
var="$(func)" || return

if [[ "${my_var}" = "some_string" ]]; then
  do_something
fi
-z (string length is zero) and -n (string length is not zero)

if ! mv "${file_list}" "${dest_dir}/" ; then
fi

Scp copy from local to remote:
scp /file/to/send username@remote:/where/to/put
Remote to local:
scp username@remote:/file/to/send /where/to/put
Send files between two remote hosts:

scp username@remote_1:/file/to/send username@remote_2:/where/to/put

Copy file from remote host to local via gateway 
scp -o "ProxyCommand ssh $USER@$bastion-host nc $destinationHost 22" $USER@$destinationHost:/home/$USER/heapdump.hprof heapdump.hprof

Copy file from local to remote host via gateway 
scp -o "ProxyCommand ssh $USER@$bastion-host nc $destinationHost 22" heapdump.hprof $USER@$destinationHost:/home/$USER/heapdump.hprof 

Netcat - nc
Listening on server
nc -l 2389 > test

nc -k -l 2389 - server would stay up
Connect to server on specific port

cat testfile | nc remoteHost 2389
Port Scanning
nc -zv remoteHost 20-30

Bulk rename files
brew install rename
rename -n -v 's/\.csv$/\.json/' *.csv
-n: --just-print/--dry-run

Brace Expansion
echo a{d,c,b}e

sleep 10 - sleep 10 seconds
wait pid - wait process finish
command &
wait $!

Check Linux System Info
lscpu
free -t -m
cat /proc/pid/smaps
pmap pid | grep total

uptime
sar
jstack -m
pstack

top
top -p PID 
-on mac: top -pid PID
top -c or Press 'c' in top view: to show full command
sort on other fields (default by cpu)

Press "SHIFT + F" and Select your choice below and press ENTER.

Get the hostname of remote server
host ip-address

Check system time zone
date
date +%Z
cat  /etc/localtime

Create zip file
gzip -k the-file
- without the tar, -k: keep the original file

tar -czf my.tar.gz the-folder_or_file
gunzip file.gz
gzip -d file.gz
unzip –t file.zip
test whether zipfile is corrupted or not

awk, gawk
gawk 'match($0, pattern, ary) {print ary[1]}'

vi
Count the number of occurrences of a word
:%s/pattern//ng
:set all
- (no)nu, (no)ic,
Compound search on multiple lines
/pattern1/;/pattern2/
Bookmark
MA -> `A -> ``

dd, 5dd
d$  - delete to end of line
d0  - delete to beginning of line

1,$d - delete all
1,.d  - delete to beginning of file
.,$d  - delete to end of file
Y - copy
- p pastes after the cursor position
- P pastes before.

5Y
Y$ - the end of the line
G - go to the last line

ZZ in command line - :wq to exit vi
append a file to current file

:r file2

Misc
mmv - Mass Move and rename
mmv "*.csv" "#1.xls"
To install mmv
yum install -y http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
yum install mmv

apropos - search the whatis database for strings
apropos "kill process"

unzip '*.zip'
unzip *.zip would fail with error: "caution: filename not matched"
gunizp *.gz - don't add ' or ""

Java
Use jstack to generate thread dump
nohup jmap -F -dump:format=b,file=/root/heapdump.hprof pid &

Commands for troubleshooting
find class in jars
find . -name "*.jar" | xargs grep Hello.class
Search Contents of .jar Files for Specific String

gfind . -iname '*.jar' -printf "unzip -c %p | grep -q 'string_to_search' && echo %p\n" | sh

Resources
Google Shell Style Guide
Post a Comment

Labels

Java (159) Lucene-Solr (110) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (37) Eclipse (33) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) Troubleshooting (13) PowerShell (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) Problem Solving (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) Lesson Learned (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts