Shortcuts:
Ctrl + a - Move to the start of line
Ctrl + e - Move to the end of line
Clear the screen: Ctrl + l
Search as you type. Ctrl + r and type the search term; Repeat Ctrl + r to loop through results.
!$
tac: print a file line by line in reverse order.
List all open ports
netstat -ltn -- all process that listens on tcp ports
-udp|-utroub
--tcp|-t
--listeningtro
--program|-p
Which process opens 9160
lsof -i :9160
lsof -p pid1
lsof -Pani -p PID
lsof /var/log/system.log
List opened files under a directory
lsof +D /var/log/
List files opened by a specific user
lsof -u user
lsof -u ^user
lsof -i
iptables
To get a complete presentation of the netfilter rules
iptables -vL -t filter
iptables -vL -t nat
iptables -vL -t mangle
iptables -vL -t raw
iptables -vL -t security
nohup someCommand > someFile.log 2>&1 &
sort | uniq -d
-u Only output lines that are not repeated in the input.
-d Only output lines that are repeated in the input.
-c count
Sort the output with the most frequent lines on top
sort FILE | uniq -c | sort -nr
find /usr -size +10M
find . -exec xxx {} \;
Find configuration data
find /etc -type f -exec grep -FHi "ENCRYPTION_PASSWORD" {} +
grep -r ENCRYPTION_PASSWORD /etc 2>/dev/null
find /etc -type f -print0 | xargs -0 grep ENCRYPTION_PASSWORD
Find biggest files
find ~/ -type f -exec du -Sh {} + | sort -rh | head -n 5
How long a command takes
time curl ""
List all functions
declare -f
declare -F - only list function names
declare -f function_name
Truncate a file
cat /dev/null > file
> file
Generate random number
/dev/random block when entropy pool is exhausted
/dev/urandom will not block
echo $RANDOM
od -An -N1 -i /dev/urandom
od -An -N2 -i /dev/urandom
for i in {1..5}; do echo $RANDOM; done
Display disk usage
du -sm
List all directories and their total size:
du -sh *
-s: Display an entry for each specified file. (Equivalent to -d 0)
Show only total for each directories
du -h -d 1
List hidden files: ls -a | grep "^\."
List files and sorted by size: ls -l | grep ^- | sort -nr
List link files: ls -l | grep '^l'
ls -l --block-size=M
Grep:
-c --count: Show count of matched lines
-E, --extended-regexp - same as egrep
-n, --line-number
-l, --files-with-matches
-L, --files-without-match
grep -E '^abc(worda|wordb)' /etc/group
-n or --line-number
-A NUM, --after-context=NUM
Print NUM lines of trailing context after matching lines.
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines.
-C NUM, --context=NUM
Print NUM lines of output context.
-w - searched as a word
-o, --only-matching
Prints only the matching part of the lines.
-l -- only show matched file name
-w -- only if it's a whole word
-r -- recursively search
grep -r --exclude=\*.{html,htm,js} pattern rootdir
Grep file that contains binary data
cat -v tmp/test.log | grep regex
-v Display non-printing characters so they are visible.
Search specific line ranges:
sed -n 'startLine,endLinep' a.txt | grep XX
Use extended regular expression with grep -E
scroll results and pagination
grep ** | more/less
sed s/word1/word2/g fileName
Only display nth line: sed -n 'n p' file
Delete nth line: sed 'n d' file > newFile
Delte nth line in place: sed –i 'n d' file.txt
Remove last line: sed –i '$ d' file.txt
-i change in place
Delete first line: sed –i '$ d' file.txt
sed –i 'm,n d' file.txt
sed -i '' 's/original/replace/' file.txt
sed -i.bak ...
sed -n 'n p' file.txt | wc -c
-i extension | --in-place
Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved.
xargs
echo a b c | xargs echo
find /tmp -name "*.bk" -type f -print | xargs /bin/rm -f
find /tmp -name "*.bk" -print0 | xargs -0 -I {} mv {} ~/bk.files
-- better: find /tmp -depth -name "*.bk" -type f -delete
find /tmp -name "*.bk" -print0 | xargs -0 -I file mv file ~/bk.files
cut -d: -f1 < /etc/passwd | sort | xargs echo
-I replstr
--null, -0 - handle spaces in file name
Change xargs to expect NUL (``\0'') characters as separators, instead of spaces and newlines. This is expected to be used in concert with the -print0 function in find
less +F the.log
Ctrl+c to normal less mode
pv - monitor the progress of data through a pipe
pv the.gz | gunzip
jq - sed for JSON
curl the_url | jq
Scp copy from local to remote:
scp /file/to/send username@remote:/where/to/put
Remote to local:
scp username@remote:/file/to/send /where/to/put
Send files between two remote hosts:
scp username@remote_1:/file/to/send username@remote_2:/where/to/put
Copy file from remote host to local via gateway
scp -o "ProxyCommand ssh $USER@$bastion-host nc $destinationHost 22" $USER@$destinationHost:/home/$USER/heapdump.hprof heapdump.hprof
Copy file from local to remote host via gateway
scp -o "ProxyCommand ssh $USER@$bastion-host nc $destinationHost 22" heapdump.hprof $USER@$destinationHost:/home/$USER/heapdump.hprof
Netcat - nc
Listening on server
nc -l 2389 > test
nc -k -l 2389 - server would stay up
Connect to server on specific port
cat testfile | nc remoteHost 2389
Port Scanning
nc -zv remoteHost 20-30
Bulk rename files
brew install rename
rename -n -v 's/\.csv$/\.json/' *.csv
-n: --just-print/--dry-run
Brace Expansion
echo a{d,c,b}e
sleep 10 - sleep 10 seconds
wait pid - wait process finish
command &
wait $!
Find out current working directory of a running process
pwdx $PID
lsof -p $PID | grep cwd
readlink -e /proc/$PID/cwd
Setting an environment variable for one command only
FOO=bar bash -c 'somecommand someargs | somecommand2'
(export FOO=bar ; somecommand someargs | somecommand2)
Change working directory for current command only
(cd SOME_PATH && exec_some_command)
Check Linux System Info
lscpu
free -t -m
cat /proc/pid/smaps
pmap pid | grep total
uptime
sar
jstack -m
pstack
top
top -p PID
-on mac: top -pid PID
top -c or Press 'c' in top view: to show full command
top -H -p $pid
sort on other fields (default by cpu)
Press "SHIFT + F" and Select your choice below and press ENTER.
Get the hostname of remote server
host ip-address
Check system time zone
date
date +%Z
cat /etc/localtime
Create zip file
gzip -k the-file
- without the tar, -k: keep the original file
tar -czf my.tar.gz the-folder_or_file
gunzip file.gz
gzip -d file.gz
unzip –t file.zip
test whether zipfile is corrupted or not
awk, gawk
gawk 'match($0, pattern, ary) {print ary[1]}'
vi
Count the number of occurrences of a word
:%s/pattern//ng
:set all
- (no)nu, (no)ic,
Compound search on multiple lines
/pattern1/;/pattern2/
Bookmark
MA -> `A -> ``
dd, 5dd
d$ - delete to end of line
d0 - delete to beginning of line
1,$d - delete all
1,.d - delete to beginning of file
.,$d - delete to end of file
Y - copy
- p pastes after the cursor position
- P pastes before.
5Y
Y$ - the end of the line
G - go to the last line
ZZ in command line - :wq to exit vi
append a file to current file
:r file2
Misc
ps -ef | grep PROCESS | grep -v grep | awk '{print $2}' | xargs kill -9
tail **/*.log
cp
cp --parents a/b/c existing_dir
--no-target-directory | -T
clear log without delete it
cp /dev/null filename
> filename
su different_user -c "command"
mmv - Mass Move and rename
mmv "*.csv" "#1.xls"
To install mmv
yum install -y http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
yum install mmv
Compare difference of two folders
diff -rq folder1 folder2
apropos - search the whatis database for strings
apropos "kill process"
unzip '*.zip'
unzip *.zip would fail with error: "caution: filename not matched"
gunizp *.gz - don't add ' or ""
Add text to the beginning of a file
echo 'task goes here' | cat - todo.txt > temp && mv temp todo.txt
Java
Use jstack to generate thread dump
nohup jmap -F -dump:format=b,file=/root/heapdump.hprof pid &
Commands for troubleshooting
find class in jars
find . -name "*.jar" | xargs grep Hello.class
Search Contents of .jar Files for Specific String
gfind . -iname '*.jar' -printf "unzip -c %p | grep -q 'string_to_search' && echo %p\n" | sh
yum -y install java8-server-jre.x86_64 --nogpgcheck yum repolist
yum-config-manager --disable \*
yum-config-manager --add-repo repo_url
sudo without password
echo 'username ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
sed -i 's/.*requiretty$/#Defaults requiretty/' /etc/sudoers
permission denied when run sudo with redirection
- Output redirection (via the > operator) is done by the shell, not by echo.
sudo bash -c "echo 'net.ipv6.conf.all.disable_ipv6 = 1' >>/etc/sysctl.conf"
Find hostname of remote ip address
host $ip
nslookup $ip
VI
:set ic|noic
Resources
Google Shell Style Guide
Ctrl + a - Move to the start of line
Ctrl + e - Move to the end of line
Clear the screen: Ctrl + l
Search as you type. Ctrl + r and type the search term; Repeat Ctrl + r to loop through results.
tac: print a file line by line in reverse order.
List all open ports
netstat -ltn -- all process that listens on tcp ports
-udp|-utroub
--tcp|-t
--listeningtro
--program|-p
Which process opens 9160
lsof -i :9160
lsof -p pid1
lsof -Pani -p PID
lsof /var/log/system.log
List opened files under a directory
lsof +D /var/log/
List files opened by a specific user
lsof -u user
lsof -u ^user
lsof -i
iptables
To get a complete presentation of the netfilter rules
iptables -vL -t filter
iptables -vL -t nat
iptables -vL -t mangle
iptables -vL -t raw
iptables -vL -t security
sort | uniq -d
-u Only output lines that are not repeated in the input.
-d Only output lines that are repeated in the input.
-c count
Sort the output with the most frequent lines on top
sort FILE | uniq -c | sort -nr
find . -exec xxx {} \;
Find configuration data
find /etc -type f -exec grep -FHi "ENCRYPTION_PASSWORD" {} +
grep -r ENCRYPTION_PASSWORD /etc 2>/dev/null
find /etc -type f -print0 | xargs -0 grep ENCRYPTION_PASSWORD
Find biggest files
find ~/ -type f -exec du -Sh {} + | sort -rh | head -n 5
How long a command takes
time curl ""
List all functions
declare -f
declare -F - only list function names
declare -f function_name
Truncate a file
cat /dev/null > file
> file
Generate random number
/dev/random block when entropy pool is exhausted
/dev/urandom will not block
echo $RANDOM
od -An -N1 -i /dev/urandom
od -An -N2 -i /dev/urandom
for i in {1..5}; do echo $RANDOM; done
Display disk usage
du -sm
List all directories and their total size:
du -sh *
-s: Display an entry for each specified file. (Equivalent to -d 0)
Show only total for each directories
du -h -d 1
List hidden files: ls -a | grep "^\."
List files and sorted by size: ls -l | grep ^- | sort -nr
List link files: ls -l | grep '^l'
ls -l --block-size=M
find . -type d -maxdepth 1 -name "H2*"
ls -d H2*/
-c --count: Show count of matched lines
-E, --extended-regexp - same as egrep
-n, --line-number
-l, --files-with-matches
-L, --files-without-match
grep -E '^abc(worda|wordb)' /etc/group
-n or --line-number
-A NUM, --after-context=NUM
Print NUM lines of trailing context after matching lines.
-B NUM, --before-context=NUM
Print NUM lines of leading context before matching lines.
-C NUM, --context=NUM
Print NUM lines of output context.
-w - searched as a word
-o, --only-matching
Prints only the matching part of the lines.
-l -- only show matched file name
-w -- only if it's a whole word
-r -- recursively search
grep -r --exclude=\*.{html,htm,js} pattern rootdir
Grep file that contains binary data
cat -v tmp/test.log | grep regex
-v Display non-printing characters so they are visible.
sed -n 'startLine,endLinep' a.txt | grep XX
Use extended regular expression with grep -E
scroll results and pagination
grep ** | more/less
sed s/word1/word2/g fileName
Only display nth line: sed -n 'n p' file
Delete nth line: sed 'n d' file > newFile
Delte nth line in place: sed –i 'n d' file.txt
Remove last line: sed –i '$ d' file.txt
-i change in place
Delete first line: sed –i '$ d' file.txt
sed –i 'm,n d' file.txt
sed -i '' 's/original/replace/' file.txt
sed -i.bak ...
sed -n 'n p' file.txt | wc -c
-i extension | --in-place
Edit files in-place, saving backups with the specified extension. If a zero-length extension is given, no backup will be saved.
xargs
echo a b c | xargs echo
find /tmp -name "*.bk" -type f -print | xargs /bin/rm -f
find /tmp -name "*.bk" -print0 | xargs -0 -I {} mv {} ~/bk.files
-- better: find /tmp -depth -name "*.bk" -type f -delete
find /tmp -name "*.bk" -print0 | xargs -0 -I file mv file ~/bk.files
cut -d: -f1 < /etc/passwd | sort | xargs echo
-I replstr
--null, -0 - handle spaces in file name
Change xargs to expect NUL (``\0'') characters as separators, instead of spaces and newlines. This is expected to be used in concert with the -print0 function in find
less +F the.log
Ctrl+c to normal less mode
pv - monitor the progress of data through a pipe
pv the.gz | gunzip
jq - sed for JSON
curl the_url | jq
Scp copy from local to remote:
scp /file/to/send username@remote:/where/to/put
Remote to local:
scp username@remote:/file/to/send /where/to/put
Send files between two remote hosts:
scp username@remote_1:/file/to/send username@remote_2:/where/to/put
scp -o "ProxyCommand ssh $USER@$bastion-host nc $destinationHost 22" $USER@$destinationHost:/home/$USER/heapdump.hprof heapdump.hprof
Copy file from local to remote host via gateway
scp -o "ProxyCommand ssh $USER@$bastion-host nc $destinationHost 22" heapdump.hprof $USER@$destinationHost:/home/$USER/heapdump.hprof
Netcat - nc
Listening on server
nc -l 2389 > test
nc -k -l 2389 - server would stay up
Connect to server on specific port
cat testfile | nc remoteHost 2389
Port Scanning
nc -zv remoteHost 20-30
Bulk rename files
brew install rename
rename -n -v 's/\.csv$/\.json/' *.csv
-n: --just-print/--dry-run
Brace Expansion
echo a{d,c,b}e
sleep 10 - sleep 10 seconds
wait pid - wait process finish
command &
wait $!
Find out current working directory of a running process
pwdx $PID
lsof -p $PID | grep cwd
readlink -e /proc/$PID/cwd
Setting an environment variable for one command only
FOO=bar bash -c 'somecommand someargs | somecommand2'
(export FOO=bar ; somecommand someargs | somecommand2)
(cd SOME_PATH && exec_some_command)
Check Linux System Info
lscpu
free -t -m
cat /proc/pid/smaps
pmap pid | grep total
uptime
sar
jstack -m
pstack
top
top -p PID
-on mac: top -pid PID
top -c or Press 'c' in top view: to show full command
top -H -p $pid
sort on other fields (default by cpu)
Press "SHIFT + F" and Select your choice below and press ENTER.
Get the hostname of remote server
host ip-address
Check system time zone
date
date +%Z
cat /etc/localtime
Create zip file
gzip -k the-file
- without the tar, -k: keep the original file
tar -czf my.tar.gz the-folder_or_file
gunzip file.gz
gzip -d file.gz
unzip –t file.zip
test whether zipfile is corrupted or not
awk, gawk
gawk 'match($0, pattern, ary) {print ary[1]}'
vi
Count the number of occurrences of a word
:%s/pattern//ng
:set all
- (no)nu, (no)ic,
Compound search on multiple lines
/pattern1/;/pattern2/
Bookmark
MA -> `A -> ``
dd, 5dd
d$ - delete to end of line
d0 - delete to beginning of line
1,$d - delete all
1,.d - delete to beginning of file
.,$d - delete to end of file
Y - copy
- p pastes after the cursor position
- P pastes before.
5Y
Y$ - the end of the line
G - go to the last line
ZZ in command line - :wq to exit vi
append a file to current file
:r file2
Misc
ps -ef | grep PROCESS | grep -v grep | awk '{print $2}' | xargs kill -9
tail **/*.log
cp
cp --parents a/b/c existing_dir
--no-target-directory | -T
clear log without delete it
cp /dev/null filename
> filename
su different_user -c "command"
mmv - Mass Move and rename
mmv "*.csv" "#1.xls"
To install mmv
yum install -y http://li.nux.ro/download/nux/dextop/el7/x86_64/nux-dextop-release-0-1.el7.nux.noarch.rpm
yum install mmv
Compare difference of two folders
diff -rq folder1 folder2
apropos - search the whatis database for strings
apropos "kill process"
unzip '*.zip'
unzip *.zip would fail with error: "caution: filename not matched"
gunizp *.gz - don't add ' or ""
Add text to the beginning of a file
echo 'task goes here' | cat - todo.txt > temp && mv temp todo.txt
Java
Use jstack to generate thread dump
nohup jmap -F -dump:format=b,file=/root/heapdump.hprof pid &
Commands for troubleshooting
find class in jars
find . -name "*.jar" | xargs grep Hello.class
Search Contents of .jar Files for Specific String
gfind . -iname '*.jar' -printf "unzip -c %p | grep -q 'string_to_search' && echo %p\n" | sh
yum -y install java8-server-jre.x86_64 --nogpgcheck yum repolist
yum-config-manager --disable \*
yum-config-manager --add-repo repo_url
sudo without password
echo 'username ALL=(ALL) NOPASSWD: ALL' >> /etc/sudoers
sed -i 's/.*requiretty$/#Defaults requiretty/' /etc/sudoers
permission denied when run sudo with redirection
- Output redirection (via the > operator) is done by the shell, not by echo.
sudo bash -c "echo 'net.ipv6.conf.all.disable_ipv6 = 1' >>/etc/sysctl.conf"
Find hostname of remote ip address
host $ip
nslookup $ip
VI
:set ic|noic
Resources
Google Shell Style Guide
