Linux Network Commands


Linux Network Commands

iptables


lsof - list open file

http://wikis.sun.com/pages/viewpage.action?pageId=49906332

http://sial.org/howto/debug/unix/lsof/

Using "lsof" in the Real World

Finding open files with lsof

-?|-h list help          -a AND selections (OR)     -b avoid kernel blocks

-c c  cmd c, /c/[bix]    +c w  COMMAND width (9)    

+d s  dir s files        -d s  select by FD set     +D D  dir D tree *SLOW?*

                       -i select IPv[46] files    -l list UID numbers

-n no host names         -N select NFS files         -o list file offset

-O avoid overhead *RISK  -P no port names           -R list paRent PID

-s list file size            -t terse listing           -T disable TCP/TPI info

-U select Unix socket    -v list version info       -V verbose search

+|-w  Warnings (+)       -X skip TCP&UDP files      -Z Z  context [Z]

-- end option scan

+f|-f  +filesystem or -file names  

-F [f] select fields; -F? for help 

+|-L [l] list (+) suppress (-) link counts < l (0 = all; default = 0)

                                    +m [m] use|create mount supplement

+|-M   portMap registration (-)       -o o   o 0t offset digits (8)

-p s   exclude(^)|select PIDs         -S [t] t second stat timeout (15)

-T qs TCP/TPI Q,St (s) info

-g [s] exclude(^)|select and print process group IDs

-i i   select by IPv[46] address: [46][proto][@host|addr][:svc_list|port_list]

+|-r [t] repeat every t seconds (15); + until no files, - forever

-u s   exclude(^)|select login|UID set s

-x [fl] cross over +d|+D File systems or symbolic Links

names  select named files or files on named file system

Examples

lsof /etc/passwd

lsof /dev/cdrom

lsof `which apache2`

Show us what files are opened by processes whose names starts by "k" (klogd, kswapd...) and bash. Show us what files are opened by init:

lsof -c k; lsof -c bash; lsof -c init

Show Listen Addresses

lsof -i

lsof list both IPv6 and IPv4 related files by default. You specified -i4 if you want entries with IPv4 only, same thing to IPv6, specified -i6.

lsof -i4 -n

lsof -i -n -a -u user

lsof -a -i -nP -c ntpd

List all opened Internet and UNIX domain files:

lsof -i –U; lsof -a -i -n -p 11108

List all opened internet sockets and sockets related to port 80: lsof -i :80

To list all files using any protocol on ports 513, 514, or 515 of host wonderland.cc.purdue.edu, use:

lsof -i @wonderland.cc.purdue.edu:513-515s

lsof -i @mace; lsof -i @192.168.1.10

lsof -p 456,123,789 -u 1234,abe

Find what process is preventing a particular file system from unmounting:

A stray process can prevent umount command from succeeding. Rather than use the -f flag and potentially cause corruptions, lsof can show you which processes need to be stopped first.


tcpdump - dump traffic on a network

http://linux.about.com/library/cmd/blcmdl8_tcpdump.htm

http://linux.byexamples.com/archives/283/simple-usage-of-tcpdump/

tcpdump [ -adeflnNOpqRStuvxX ] [ -c count ] [ -C file_size ] [ -F file ] [ -i interface ] [ -m module ] [ -r file ] [ -s snaplen ] [ -T type ] [ -U user ] [ -w file ] [ -E algo:secret ][ expression ]

-i Listen on interface. If unspecified, tcpdump searches the system interface list for the lowest numbered, configured up interface (excluding loopback).

-c Exit after receiving count packets.

-r

Read packets from file (which was created with the -w option). Standard input is used if file is ``-''.

-v,-vv,-vvv erbose output.

Expression

dst port port True if the packet is ip/tcp, ip/udp, ip6/tcp or ip6/udp and has a destination port value of port.

src port port True if the packet has a source port value of port.

port port True if either the source or destination port of the packet is port. Any of the above port expressions can be prepended with the keywords, tcp or udp, as in:

tcp src port port which matches only tcp packets whose source port is port.

net net True if either the IPv4/v6 source or destination address of the packet has a network number of net.

examples:

tcpdump host sundown -i eth2

tcpdump -w test.pcap -i eth2 tcp port 6881

tcpdump -w test.pcap -i eth1 tcp port 6881 or udp \( 33210 or 33220 \)

To print traffic between helios and either hot or ace:

tcpdump host helios and \( hot or ace \)

To print all IP packets between ace and any host except helios:

tcpdump ip host ace and not helios

To print traffic neither sourced from nor destined for local hosts (if you gateway to one other net, this stuff should never make it onto your local net).

tcpdump ip and not net mtc037c0.storage.tucson.ibm.com

To print the start and end packets (the SYN and FIN packets) of each TCP conversation that involves a non-local host.

tcpdump 'tcp[tcpflags] & (tcp-syn|tcp-fin) != 0 and not src and dst net localnet'

To print IP packets longer than 576 bytes sent through gateway snup:

tcpdump 'gateway snup and ip[2:2] > 576'

To print IP broadcast or multicast packets that were not sent via ethernet broadcast or multicast:

tcpdump 'ether[0] & 1 = 0 and ip[16] >= 224'

To print all ICMP packets that are not echo requests/replies (i.e., not ping packets):

tcpdump 'icmp[icmptype] != icmp-echo and icmp[icmptype] != icmp-echoreply'

By default the sniff size of packets is 96 bytes, you somehow can overload that size by specified with -s.

tcpdump -w test.pcap -s 1550 dst 10.168.28.22 and tcp port 22

Some version of tcpdump allows you to define port range.

tcpdump tcp portrange 20-24

tcpdump -i eth3 dst 224.0.0.1

tcpdump -n tcp port \(1720 or 1732\) and host \(172.16.1.101 or 172.17.1.101\)


netstat

-a,-all Show the state of all sockets, not just active ones.

-c,--continuous Display information continuously, refreshing once every second.

-e, --extend Display additional information.

-i Include statistics for network devices.

-l, --listening Show only listening sockets. (These are omitted by default.)

-n,--numeric Show network addresses as numbers.

-o Include additional information such as username.

-p, --program Show the PID and name of the program to which each socket belongs

-r,--route Show routing tables.

--statistics , -s Display summary statistics for each protocol.

-t,--tcp List only TCP sockets.

-u,--udp List only UDP sockets.

-w,--raw List only raw sockets.

-x List only Unix domain sockets.

Proto Recv-Q Send-Q Local Address Foreign Address State

tcp 0 0 vhost:32803 LocalHost:smtp TIME_WAIT

tcp 0 0 vhost:32803 google.com:http ESTABLISHED

Recv-Q -Specifies the Number of Bytes which are not recevied.

Send-Q -Specifies the Number of Bytes not send to destination.

State this can be one of several values:

ESTABLISHED The socket has an established connection.

SYN_SENT The socket is actively attempting to establish a connection.

SYN_RECV A connection request has been received from the network.

FIN_WAIT1 The socket is closed, and the connection is shutting down.

FIN_WAIT2 Connection is closed, and the socket is waiting for a shutdown from the remote end.

TIME_WAIT The socket is waiting after close to handle packets still in the network.

CLOSED The socket is not being used.

CLOSE_WAIT The remote end has shut down, waiting for the socket to close.

LAST_ACK The remote end has shut down, and the socket is closed. Waiting for acknowledgement.

LISTEN The socket is listening for incoming connections. Such sockets are not included in the output unless you specify the --listen-ing (-l) or --all (-a) option.

CLOSING Both sockets are shut down but we still don鈥檛 have all our data sent.

UNKNOWN The state of the socket is unknown.

netstat --tcp --numeric

netstat --tcp --listening --programs

netstat --route displays the routing table.

netstat --statistics

netstat --statistics --raw

netstat --statistics --tcp

netstat -na| egrep '1720|Proto'

netstat -rn


traceroute [options] host [packetsize]

-n Show numerical addresses; do not look up hostnames.

traceroute 172.16.1.101

route [option] [command]

Route manipulates the kernel IP routing tables. Its primary use is to set up static routes to specific hosts or networks via an interface after it has been configured with the ifconfig(8) program.

When the add or del options are used, route modifies the routing tables. Without these options, route displays the current contents of the routing tables.

route [-v] [-A family] add [-net|-host] target [netmask Nm] [gw Gw] [metric N] [mss M] [window W] [irtt I] [reject] [mod] [dyn] [reinstate] [[dev] If]

route [-v] [-A family] del [-net|-host] target [gw Gw] [netmask Nm] [metric N] [[dev] If]

-n show numerical addresses instead of trying to determine symbolic host names.

del delete a route.

add add a new route.

target the destination network or host. You can provide IP addresses in dotted decimal or host/network names.

-net the target is a network.

-host the target is a host

netmask NM when adding a network route, the netmask to be used.

route add -net 127.0.0.0 adds the normal loopback entry, using netmask 255.0.0.0 and associated with the "lo" device.

route add -net 192.56.76.0 netmask 255.255.255.0 dev eth0

adds a route to the network 192.56.76.x via "eth0".

route add default gw mango-gw

adds a default route (which will be used if no other route matches). All packets using this route will be gatewayed through "mango-gw". The device which will actually be used for that route depends on how we can reach "mango-gw" - the static route to "mango-gw" will have to be set up before.

route add ipx4 sl0

Adds the route to the "ipx4" host via the SLIP interface (assuming that "ipx4" is the SLIP host).

route add -net 192.57.66.0 netmask 255.255.255.0 gw ipx4

This command adds the net "192.57.66.x" to be gatewayed through the former route to the SLIP interface.

route add -net 224.0.0.0 netmask 224.0.0.0 eth0

route add -net 224.0.0.0 netmask 224.0.0.0 eth3


route delete -net 224.0.0.0 netmask 224.0.0.0 eth0

route delete -net 224.0.0.0 netmask 224.0.0.0 eth3

route -e


arp

Clear, add to, or dump the kernel's ARP cache, the arp command displays and modifies the Internet-to-adapter address translation tables used by the Address in Networks and communication management. The arp command displays the current ARP entry for the host specified by the HostName variable. The host can be specified by name or number, using Internet dotted decimal notation.

-t type Search for type entries when examining the ARP cache. type must be ether (Ethernet) or ax25 (AX.25 packet radio);

-a [hosts] Display hosts' entries or, if none are specified, all entries.

-d host Remove host's entry.

-s host hardware-address

Add the entry host hardware-address, where ether class addresses are 6 hexadecimal bytes, colon-separated.

-f file Read entries from file and add them.

1 To add a single entry to the arp mapping tables until the next time the system is restarted, type:

arp -s 802.3 host2 0:dd:0:a:85:0 temp

2 To delete a map table entry for the specified host with the arp command, type: arp -d host1 flag

3 To display arp entries for atm host host1 , type: arp -t atm -a host1

4 To add a PVC arp entry for atm host host2, type: arp -t atm -s atm pvc 0:20 host2

5 To add a PVC arp entry for an interface at0, type: arp -t atm -s atm pvc 0:20 if at0


ifconfig [interface]

ifconfig [interface address_family parameters addresses]

interface String of the form name unit, for example, en0.

Arguments

address_family

Since an interface may receive transmissions in differing protocols, each of which may require separate naming schemes, you can specify the address_family to change the interpretation of the remaining parameters. You may specify inet (the default; for TCP/IP), ax25 (AX.25 Packet

Radio), ddp (Appletalk Phase 2), or ipx (Novell).

Parameters

broadcast

(inet only.) Specify address to use to represent broadcasts to the network. Default is the address with a host part of all 1s (i.e.,x.y.z.255 for a class C network).

dest_address

Specify the address of the correspondent on the other end of a

point-to-point link.

down Mark an interface "down" (unresponsive).

hw class address

Set the interface's hardware class and address. class may be ether (Ethernet), ax25 (AX.25 Packet Radio), or ARCnet.

netmask mask

(inet only.) Specify how much of the address to reserve for subdividing networks into subnetworks. mask can be specified as a single hexadecimal number with a leading 0x, with a dot notation Internet address, or with a pseudonetwork name listed in the network table /etc/networks.

pointopoint/-pointopoint [address]

Enable/disable point-to-point interfacing, so that the connection between the two machines is dedicated.

up Mark an interface "up" (ready to send and receive).

ifconfig -a

Configuring an interface

ifconfig eth0 192.168.0.1 netmask 255.255.255.0 broadcast 192.168.0.255 up

ifconfig eth1 down/up

To add a second IP address to wlan0:

ifconfig wlan0:1 192.168.2.41 netmask 255.255.255.0

To change the hardware address (MAC address) assigned to eth0 (useful when setting up a router for a DSL or cable modem):

ifconfig eth0 hw ether 01:02:03:04:05:06

ifup - bring a network interface up

ifdown - take a network interface down

Examples: ifup -a, ifup eth0, ifdown -a

host host www.google.com

Give a host name and the command will return IP address. Unlike nslookup, the host command will use both /etc/hosts as well as DNS.

nslookup - Give a host name and the command will return IP address.

dig - DNS lookup utility

dig www.google.com

Network IP aliasing:

Assign more than one IP address to one ethernet card:

ifconfig eth0 XXX.XXX.XXX.XXX netmask 255.255.255.0 broadcast XXX.XXX.XXX.255

ifconfig eth0:0 192.168.10.12 netmask 255.255.255.0 broadcast 192.168.10.255

ifconfig eth0:1 192.168.10.14 netmask 255.255.255.0 broadcast 192.168.10.255

route add -host XXX.XXX.XXX.XXX dev eth0

route add -host 192.168.10.12 dev eth0

route add -host 192.168.10.14 dev eth0

In this example 0 and 1 are aliases in addition to the regular eth0.

Changing the host name:

Issue the command: hostname new-host-name

Change network configuration file: /etc/sysconfig/network, edit entry: HOSTNAME=new-host-name

Restart systems which relied on the hostname (or reboot):

Restart network services: service network restart (or: /etc/init.d/network restart)

ethtool - Display or change ethernet card settings

ethtool eth0

ethtool -r eth0

-r --negotiate restarts auto-negotiation on the specified ethernet device, if auto-negotiation is enabled.

ethtool en0 |grep Speed

-t –test executes adapter selftest on the specified ethernet device

insmod filename [module-options]

System administration command. Load the module filename into the kernel. Simpler but less flexible than the modprobe command.

modprobe [options] [modules]

System administration command. With no options, attempt to load the specified module, as well as all modules on which it depends. If more than one module is

specified, attempt to load further modules only if the previous module failed to load.

-a Load all listed modules, not just the first one.

-l [pattern] List all existing modules.

-r Remove the specified modules, as well as the modules on which they depend.

-t type Load only a specific type of module. Consult /etc/conf.modules for the

directories in which all modules of that type reside.

Related: /sbin/insmod, /sbin/rmmod, /sbin/depmod

dmesg is used to examine or control the kernel ring buffer.

dmesg [ -c ] [ -n level ] [ -s bufsize ]

-sbufsize

Use a buffer of size bufsize to query the kernel ring buffer. This is 16392 by default.

-nlevel

Set the level at which logging of messages is done to the console. For example, -n 1 prevents all messages, expect panic messages, from appearing on the console. All levels of messages are still written to /proc/kmsg, so syslogd(8) can still be used to control exactly where kernel messages appear. When the -n option is used, dmesg will not print or clear the kernel ring buffer.

dmesg | grep -i usb

dmesg | grep -i tty

dmesg | grep -i memory

dmesg | grep -i dma

The output of dmesg is maintained in the log file /var/log/dmesg.


Configuration Files:

/etc/dhcpd.conf

/etc/hosts - locally resolve node names to IP addresses

/etc/resolv.conf - host name resolver configuration file

search name-of-domain.com - Name of your domain or ISP's domain if using their name server

nameserver XXX.XXX.XXX.XXX - IP address of primary name server

nameserver XXX.XXX.XXX.XXX - IP address of secondary name server

This configures Linux so that it knows which DNS server will be resolving domain names into IP addresses. If using DHCP client, this will automatically be sent to you by the ISP and loaded into this file as part of the DHCP protocol. If using a static IP address, ask the ISP or check another machine on your network.


Resources:

http://www.hscripts.com/tutorials/linux-commands/netstat.html

http://www.yolinux.com/TUTORIALS/LinuxTutorialNetworking.html


Bug fixing


Bug fixing

Lesson 1:
In the program, Some threads are designed to run forever until program is shutdown, but unfortunately, they didn't capture all-possibly-thrown exceptions,
and this would cause thread exit unexpectedly.

Lesson 2: When program runs slowly or weirdly, check system status, and all possibility, and guess reasonably.


Some tests defects look very weird, the program can not send out multicast messages intermittently. At first, I guess it may be code problem, or because we have upgraded machine
to new operating system, new JDK, so maybe new OS or new JDK is the culprit. when I do test, I found that when I hit the problem, the command 'java -version' would hang for ever. But at that moment, I ignore this obvious information.
At last my colleague figure out the root cause of the problem, that is because one process in the machine consumes too many system resource,
which cause all other processes to starve and frozen, and run extremely slow.
UID PID PPID C STIME TTY TIME CMD
root 307694 1 50 15:14:20 - 33:08 /process_cmd
C

(-f, l, and -l flags) CPU utilization of process or thread, incremented each time the system clock ticks and the process or thread is found to be running. The value is decayed by the scheduler by dividing it by 2 once per second.
For the sched_other policy, CPU utilization is used in determining process scheduling priority. Large values indicate a CPU intensive process and result in lower process priority whereas small values indicate an I/O intensive process and result in a more favorable priority.

How stupidly I didn't use ps and top command to check system run status, and ignore when I discover 'java -version' hang, and didn't catch the connection.


Java Heapdump


Java Heapdump

Generate Heapdump for Sun JDK:
-XX:+HeapDumpOnOutOfMemoryError
Generate Heapdump for IBM JDK 1.4
http://publib.boulder.ibm.com/infocenter/javasdk/v6r0/index.jsp?topic=/com.ibm.java.doc.diagnostics.60/html/env_jvm.html
set IBM_HEAP_DUMP=true
set IBM_HEAPDUMP=true
set IBM_HEAPDUMP_OUTOFMEMORY=true
set IBM_JAVADUMP_OUTOFMEMORY=true
set IBM_JAVA_HEAPDUMP_TEXT=true


IBM_HEAPDUMP or IBM_HEAP_DUMP:
Setting either of these to any value (such as true) enables heap dump production by means of signals.
IBM_HEAPDUMP_OUTOFMEMORY:
When set to TRUE or 1 - generates a heapdump each time an out-of-memory exception is thrown, even if it is handled.
IBM_JAVADUMP_OUTOFMEMORY:
When set to TRUE or 1 - generates a Javadump each time an out-of-memory exception is thrown, even if it is handled.
IBM_HEAPDUMPDIR:
Specify an alternative location for Heapdump files.
IBM_JAVACOREDIR:
Specify an alternative location for Javadump files

IBM_JAVA_HEAPDUMP_TEXT:
Use this environment variable to cause the JVM to generate a text (human readable) Heapdump.

JAVA_DUMP_OPTS:
Use this environment variable to control the conditions under which Heapdumps (and other dumps) are produced.

The recommended way of controlling the production of diagnostic data is the -Xdump command-line option. However, if you do use JAVA_DUMP_OPTS, the recommended default value is:
JAVA_DUMP_OPTS="ONERROR (JAVADUMP,SYSDUMP) ONEXCEPTION (JAVADUMP,SYSDUMP), ONDUMP (JAVADUMP)"
see Using Heapdump
Using Heapdump
Heapdump is an IBM JVM facility that generates a dump of all the reachable objects that are on the Java heap. It shows the objects that are using large amounts of memory on the Java heap, and what is preventing them from being collected by the Garbage Collector.


You can generate a Heapdump in either of two ways:
Explicit generation
JVM-triggered generation
Explicit generation of a Heapdump
You can explicitly generate a Heapdump in either of the following ways:
By sending a signal to the JVM from the operating system
By using the HeapDump() method inside Java code that is being executed


For Linux and AIX, send the JVM the signal SIGQUIT (kill -QUIT or -3, or Ctrl+\ in the console window): kill -3 proceessID
For Windows, generate a SIGINT (press the Ctrl+Break keys simultaneously).
You can explicitly request a Heapdump from a Java method. The class com.ibm.jvm.Dump contains a static HeapDump() method that causes Java code to initiate a Heapdump, provided that the IBM_HEAPDUMP environment variable is set.
Triggered generation of a Heapdump
The following events automatically trigger the JVM to produce a Heapdump:
A fatal native exception occurs in the JVM (not a Java Exception)
An OutOfMemory or heap exhaustion condition occurs (optional)
Location of the generated Heapdump
The location that is specified by the IBM_HEAPDUMPDIR environment variable,if set (_CEE_DMPTARG on z/OS).
The current working directory of the JVM processes
The location that is specified by the TMPDIR environment variable, if set
The /tmp directory (X:\tmp for Windows, where X is the current working drive)
Format of Heapdump filenames
Windows heapdump.YYYYMMDD.HHMMSS.PID.txt
Linux & AIX heapdumpPID.TIME.txt
z/OS HEAPDUMP.YYYYMMDD.HHMMSS.PID.txt
Note: PID is the process ID. TIME is the number of seconds since 1/1/1970
Using VerboseGC to obtain heap information
Use the VerboseGC utility to obtain information about the Java Object heap in real time while running your Java applications. To activate this utility, run Java with the -verbosegc option:
java -verbosegc

JVM environment settings -- general options
Heap size
The maximum heap size is controlled by the -Xmx option.
The initial size of the heap is controlled by the -Xms option.
CLASSPATH
A list of directories for the JVM to find user class files, paths, or both to individual Java archive or .compressed files that contain class files

Heapdump or JavaCore Analyze Tools:
Eclipse Memory Analyzer (MAT)
The Eclipse Memory Analyzer is a fast and feature-rich Java heap analyzer that helps you find memory leaks and reduce memory consumption.
IBM HeapAnalyzer
HeapAnalyzer analyzes Java heap dumps by parsing the Java heap dump, creating directional graphs, transforming them into directional trees, and executing the heuristic search engine.
IBM Thread and Monitor Dump Analyzer for Java:jca
A tool that allows identification of hangs, deadlocks, resource contention, and bottlenecks in Java threads.


SSH Puzzles


SSH Puzzles

Recently, when I tried to ssh to remote machine, it failed and reported the following error:

$ ssh -l $USER $REMOTE_MACHINE

ssh_exchange_identification: Connection closed by remote host


and I add -v parameter to print debugging messages about its progress.

$ ssh -v -l $USER $REMOTE_MACHINE

OpenSSH_5.1p1, OpenSSL 0.9.8i 15 Sep 2008

debug1: Connecting to $REMOTE_MACHINE port 22.

debug1: Connection established.

debug1: identity file ~/.ssh/identity type -1

debug1: identity file ~/.ssh/id_rsa type 1

debug1: identity file ~/.ssh/id_dsa type -1

debug1: ssh_exchange_identification:


debug1: ssh_exchange_identification:


debug1: ssh_exchange_identification: Error: Must authenticate before using this service.


ssh_exchange_identification: Connection closed by remote host


Then my colleague told me that I should pass the intranet firewall, and after I passed the firewall, ssh worked.

So stupid to forget passing firewall first.


Labels

adsense (5) Algorithm (69) Algorithm Series (35) Android (7) ANT (6) bat (8) Big Data (7) Blogger (14) Bugs (6) Cache (5) Chrome (19) Code Example (29) Code Quality (7) Coding Skills (5) Database (7) Debug (16) Design (5) Dev Tips (63) Eclipse (32) Git (5) Google (33) Guava (7) How to (9) Http Client (8) IDE (7) Interview (88) J2EE (13) J2SE (49) Java (186) JavaScript (27) JSON (7) Learning code (9) Lesson Learned (6) Linux (26) Lucene-Solr (112) Mac (10) Maven (8) Network (9) Nutch2 (18) Performance (9) PowerShell (11) Problem Solving (11) Programmer Skills (6) regex (5) Scala (6) Security (9) Soft Skills (38) Spring (22) System Design (11) Testing (7) Text Mining (14) Tips (17) Tools (24) Troubleshooting (29) UIMA (9) Web Development (19) Windows (21) xml (5)