Notes on RMC

Notes on RMC

Introduction

Resource Monitoring and Control (RMC) is a function that gives you the ability to monitor the state of system resources and respond when predefined thresholds are crossed, so that you can perform many routine tasks automatically.

RMC is a no charge feature of AIX 5L Version 5.1 that can be configured to monitor resources (disk space, CPU usage, processor status, application processes, and so on) and perform an action in response to a defined condition.


The predefined conditions are ready to use and just need to be enabled or configured to match the exact requirements of your systems. If predefined conditions do not satisfy your systems requirements, RMC allows you to create new conditions, responses, and actions to tailor the system to respond when and how you require.

Technically, Resource Monitoring and Control (RMC) is a subset function of Reliable Scalable Cluster Technology (RSCT).

Clustered environment

RMC is capable of working in a stand-alone1 or clustered environment. In AIX 5L Version 5.1, nodes in a cluster may be configured for either of the following cluster domains: Peer domain or Management domain.

In a peer domain, all nodes are considered equal and any node can monitor and control (or be monitored and controlled) by any other node. In a management domain, a management node is aware of all nodes it is managing but the nodes

themselves know nothing of each other.

Management domain

In a management domain, nodes are managed by a management server. The management server is aware of all the nodes it manages and all managed nodes are aware of their management server. However, the managed nodes know

nothing about each other.

Filesets and packages

cd /usr/sys/inst.data/sys_bundles; grep -i rsct *

man lslpp

Lists installed software products: lslpp -L

To list the fileset that owns installp, type: lslpp -w /usr/sbin/installp

1. Stop RMC daemons: # /usr/sbin/rsct/bin/rmcctrl -z

2. Reconfigure necessary information and start RMC daemons:# /usr/sbin/rsct/bin/rmcctrl -A

Uninstall RMC from your system

To remove all the RMC filesets:# installp -ug rsct.*

Architecture and components

Provide single monitoring and management infrastructure for clusters.

Provide global access to subsystems and resources throughout the cluster.

Support operations for configuring, monitoring, and controlling all cluster resources by RMC clients.

Encapsulate all resource dependent operations.

Provide a common access control mechanism across all resources.

Support integration with other subsystems to achieve the highest levels of availability.

Resource, resource class, and attribute

Resource

The resource is a fundamental concept in the RMC architecture. A resource is an abstraction of an instance of a physical or logical entity that provides services to applications or system components. A system or cluster is composed of

numerous resources of various types.

Resource class

A resource class is a collection of resources that have similar characteristics. The resource class provides descriptive information about the properties and characteristics that are common to any resource within the resource class.

Attribute

A resource class and a resource have several attributes. An attribute has a value and a unique name within the resource class or the resource. A resource attribute is classified into either a public or a private property. The property is used as a hint for the RMC client for whether the attribute is to be presented to general users. Private attributes typically contain information that is not relevant to general users; therefore, the private attributes are hidden by default. However, you can display private attributes by specifying a flag in the command.

Attributes fall into two categories: persistent and dynamic.

Persistent attributes

For resources, persistent attributes are configuration parameters and are set either by the resource monitor harvesting real resources, or through the mkrsrc or the chrsrc commands. Persistent attributes define the characteristics of the

resource. For resource classes, persistent attributes describe or control the operations of the class.

Dynamic attributes

Dynamic attributes reflect internal states or performance variables of resources and resource classes.

You generally refer to dynamic attributes to define the monitoring condition of the resource you want to monitor.

Resource managers

Each resource manager is the interface between the RMC subsystem and a specific aspect of the AIX instance it is controlling. All resource managers have the same architecture and interact with the other RMC components.

Four groups of resource managers:

Logging and debugging >>> Audit Log resource manager

Configuration >>> configuration resource manager

Reacting to events >>> Event Response resource manager

Monitoring data

>>> File system resource manager

>>> Host resource manager

>>> Sensor resource manager

Event Response resource manager (ERRM)

The Event Response resource manager provides the system administrator with the ability to define a set of conditions to monitor in the various nodes of the cluster, and to define actions to take in response to these events. The conditions

are applied to dynamic properties of any resources of any resource manager in the cluster. The Event Response resource manager provides a simple automation mechanism for implementing event driven actions.

RMC command line interface

Environment variables

RMC is a distributed environment, where commands can be executed on any node in a cluster to act on any node in the cluster. Two environment variables, CT_CONTACT and CT_MANAGEMENT_SCOPE, control the behavior of all

RMC commands with regard to the geographical aspects of RMC.

CT_CONTACT

This variable defines where the command will be executed. It should contain the host name or the IP address of the host where the command will be executed. If you are not logged on the target node, the CLI contacts the RMC daemon on the target host specified by CT_CONTACT and passes it the commands to be executed. If the variable is unset, the default is to execute the command on the local node.

CT_MANAGEMENT_SCOPE

This variable specifies whether commands apply to only local resources or to resources located on the other nodes in the cluster. This variable can be set to an integer value of 0, 1, 2, or 3.

0 or 1 Resources on local node only

2 Resources on all the nodes in a peer domain

3 Resources on all the nodes in a management domain

If the CT_MANAGEMENT_SCOPE variable is not set to any value, the default value is 0, therefore commands refer to resources on the local node only.

Command flags and pattern match operators

Most RMC commands accept the following commonly used command flags:

Display format -l, -V, -t, -x, -d, and -D

Help -h; Selection -s

The -s flag takes a selection string argument that can contain at least one expression.

Display format

Most commands support the same flags to specify the output format. The -l and -V flags are used for interactive purpose, and display respectively long and verbose results. The -t, -x, -d, or -D flags are more commonly used within scripts, yielding results in tabular format, without header, with predefined (colon) or user specified field delimiter, that are easier to parse than the default results.

lsrsrc -V IBM.Host NodeNameList NumProcessors RealMemSize OSName

Selection

The selection string is an expression that is evaluated against each resource or class. The selection string is made of variables, operators and constants. The variables refer to persistent attributes of the target resource or class. Selection

cannot be performed on dynamic attributes.

lsrsrc -s 'Mount == "true"' IBM.FileSystem Name PercentTotUsed

lsrsrc -s 'Mount=="true" && Size < 40000' IBM.FileSystem Name PercentTotUsed Size

Pattern match operators

There are two pattern match operators, =~ and =?, and two not pattern match operators, !~ and ~?. The =~ and !~ operators match substrings, while the SQL-like =? and !? operators match full strings.

When using the extended regular expression operators =~ and !~ with wild cards, you should use:

The dot (.) to match exactly one character.

The star (*) to match zero or more occurrences of the preceding characters.

With the SQL-like syntax operators, =~ and !~, you should use the following wild card characters:

The percent sign (%) matches zero or more characters.

The underscore (_) matches exactly one character.

The percentage and underscore characters can be quoted with the pound sign (#) to override their special meaning.

lsrsrc -s 'Mount =? "t%"' IBM.FileSystem Name Mount

lsrsrc -s 'Mount =~ "t.*"' IBM.FileSystem Name Mount

A peer domain cluster setup

Preparing your node security: preprpnode svr02 svr04

Creating the cluster

To create a peer domain cluster, issue mkrpdomain on one node that is already prepared with preprpnode using either of the following methods (PD is the cluster name): mkrpdomain PD svr02 svr04

To check the status of the domain, use lsrpdomain to list all known domains and provide a summary of their status:

Bringing the cluster online: # startrpdomain PD

Adding a node to the cluster

Once the security environment is configured, the next step is to add the node to the online cluster. The command addrpnode performs this operation, and must be run on a node that is online in the cluster you are modifying. Nodes may be defined in several clusters simultaneously, but can only be online to one cluster at any time.

# addrpnode svr05; lsrpdomain; lsrpnode

Bringing a node online: # startrpnode svr05

Stopping (offline) a node in the cluster: # stoprpnode svr05

Removing a node from the cluster

If you must remove a node from the cluster, the node must first be in the offline state, and then it can be removed with the rmrpnode command. This must be executed on an online node of the cluster: # rmrpnode svr05

Stopping (offline) a cluster

The stoprpdomain command is used to take all the online nodes of a cluster offline. It must be run on an online node.

# stoprpdomain PD

The cluster configuration is preserved after the cluster status is changed to offline.

Removing a cluster

If a cluster must be deleted and the configuration information removed, rmrpdomain can be used.

To remove a peer domain cluster, do the following:

1. Bring the peer domain online, if it is not online yet.

2. Execute the rmrpdomain command on an online node: # rmrpdomain PD

3. If there is any node that is not network accessible when the peer domain is being removed, execute the rmrpdomain command with the -f option on each node to clean up the cluster configuration on the node: # rmrpdomain -f PD

SRC commands

AIX has the system resource controller (SRC), which provides a consistent controlling method for various system daemon processes (referred to as subsystems).

lssrc

The lssrc command provided by AIX SRC can be used to check if the RMC subsystems are active: # lssrc -g rsct; # lssrc -g rsct_rm

The ctrmc subsystem, which we refer to as the RMC subsystem in this redbook, is instantiated as the rmcd daemon (/usr/sbin/rsct/bin/rmcd).

A security subsystem, ctcas, is in charge of authentication in the cluster, by means of UNIX identity-based credentials. The ctcas subsystem is instantiated as the ctcasd daemon (/usr/sbin/rsct/bin/ctcasd).

Both the ctrmc and ctcas subsystems belong to the rsct group.

Other SRC commands

Even if RMC components are subsystems that can be managed with any AIX System Resource Controller (SRC) commands, we recommend you only use the SRC lssrc command with these subsystems. Do not use the startsrc and

stopsrc commands to start and stop the RMC daemon, but rather the rmcctrl (RMC Control) command.

The refresh command may be used to force the RMC subsystem to re-read its configuration files.

RSCT commands

The rmcctrl command manages both the RMC subsystem and the resource managers subsystem.

During the AIX installation, the rmcctrl-a command is performed automatically, and the following entry will be added in the /etc/inittab: # grep ctrmc /etc/inittab

ctrmc:2:once:/usr/bin/startsrc -s ctrmc > /dev/console 2>&1

Therefore, the RMC subsystem is started by default upon every reboot. There is no need to perform any action to manually start it in normal operation.

The -s and -k flags start and stop the ctrmc only.

Both the -K and -z options stops RMC and the resource managers, but the former is asynchronous while the later is synchronous. With -z, the command will not return before all subsystems have actually stopped

RMC commands

There are five categories of generic RMC commands:

Creation mkrsrc;

Display lsrsrc, lsrsrcdef, and lsactdef

Modification chrsrc

Deletion rmrsrc

Refresh refrsrc

lsrsrc

The lssrc command sends a request to the System Resource Controller to get status on a subsystem, a group of subsystems, or all subsystems.

To Get All Status lssrc [ -h Host ] -a

To Get Group Status lssrc [ -h Host ] -g GroupName

To Get Subsystem Status lssrc [ -h Host ] [ -l ] -s Subsystem

To Get Status by PID lssrc [ -h Host ] [ -l ] -p SubsystemPID

To Get Subserver Status lssrc [ -h Host ] [ -l ] -t Type [ -p SubsystemPID ] [ -o Object ] [ -P SubserverPID ]

To get the status of the tcpip subsystem group, enter: lssrc -g tcpip

You should verify that some sub-systems are running on your system: # lssrc -g rsct; lssrc -g rsct_rm

The lsrsrc command displays the persistent and dynamic attributes and their values for a resource or a resource class.

To list the names of all of the resource classes: lsrsrc

2 To list the persistent attributes for resource IBM.Host that have 4 processors, enter: lsrsrc -s "NumProcessors == 4" -A p -p 0 IBM.Host

3 To list the public dynamic attributes for resource IBM.Host on node 1, enter: lsrsrc -s 'Name == "c175n05.ppd.pok.ibm.com"' -A d IBM.Host

4 To list the Name, Variety, and ProcessorType attributes for the IBM.Processor resource on all the online nodes, enter: lsrsrc IBM.Processor Name Variety ProcessorType

5 To list both the persistent and dynamic attributes for the resource class IBM.Condition:lsrsrc -c -A b -p 0 IBM.Condition; lsrsrc IBM.Host Name

The first argument of lsrsrc is a resource class name. By default, the result will relate to resources in this class. You need to specify the -c flag to retrieve class attributes and values.

# lsrsrc -c IBM.PagingDevice; # lsrsrc IBM.PagingDevice

Attributes are either persistent (static) or dynamic. If you do not explicitely specify attribute(s) in the command line, the default is to display only persistent attributes. The -a flag followed by p (persistent), d (dynamic) or b (both) overrides

the default behavior: # lsrsrc -ad IBM.PagingDevice

By default, lsrsrc displays only public attributes. In general, attributes which are not public are only useful for application developers and not for end users. You need to specify -p0 to list all attributes: lsrsrc -p0 IBM.PagingDevice

lsrsrcdef

The lsrsrcdef command returns the description of a resource class or a resource and their attributes.

You can use it when:

Creating selection strings

Developing new monitoring conditions

Obtain detailed information about a resource

The -ap or -ad flags instruct the command to return persistent or dynaic attributes respectively

The -c flag indicates whether lsrsrcdef returns information pertaining to the class or the resource.

The -e flag indicates that the returned information will return the full description of an attribute. This description can be long and is not displayed by default.

# lsrsrcdef -ap -e IBM.FileSystem Bf

The lssrcdef command also displays public attributes only by default. Using the -p0 flag, all attributes are displayed.

refrsrc

refresh forces the resource manager owning this class to detect any changes in the configuration of resources in this class. The lsrsrc -x command returns all the resource class names.

To chop the double-quote characters returned by lsrsrc around the resource class names:

# for i in `lsrsrc -x | sed 's/\"//g'`; do; echo $i; done

refresh all resource classes using refrsrc:

# for i in `lsrsrc -x | sed 's/\"//g'`; do; refrsrc -V $i; done

Sensor commands

When you want to monitor a part of the AIX environment for which RMC does not provide a predefined resource class or attribute, you can create your own resource, called a sensor, using the Sensor resource manager. A sensor is a command that will be periodically executed, returning a set of values that can be monitored with the Event Response resource manager.

The Sensor resource manager provides four commands to list, create, modify or remove sensors; lssensor, mksensor, chsensor, and rmsensor.

lssensor

The lssensor command displays either a list of sensors, or the list of attributes of sensors. With no flags, the lssensor list the sensors defined locally, and with the -a flags, all sensors defined in the cluster. The CT_CONTACT variable indicate where the command will be executed.

If you prefer not to modify the CT_CONTACT variable, you can use the -n flag to specify the nodes on which you look for the defined sensors: # lssensor -n svr04,svr05

mksensor

The mksensor command creates a sensor on one node. You cannot create the same sensor on multiple nodes at the same time.

# lssensor -n svr05; # mksensor -n svr05 NumUsers1 "/SharedTools/NumLogins.ksh"; # lssensor -n svr05

The only option that can be set during the creation of the sensor is the period at which the sensor is executed, using the -i flag followed by an integer representing a time in seconds. Apart from the sensor name, you must specify as the second argument of the mksensor command the name of the program you want to execute. You can also specify arguments to this program by surrounding the program name and its arguments with double quotes.

# mksensor -i 300 NumUsers1 "/SharedTools/NumLogins.pl abc"

chsensor

The chsensor command is used to rename a sensor. You cannot use it to modify the sensor period or the command to be executed. If you need such a change, the only solution is to delete the sensor and recreate it.

Using the -a flag, you can rename a sensor on all nodes in the cluster, or you can restrict the renaming to a set of nodes with the -n flag, followed by the nodes names.

rmsensor

The rmsensor command delete sensor on one or several nodes at the same time, using the same -a and -n flags as chsensor. You can delete several sensors in one command by passing several sensor names as arguments to rmsensor.

ERRM commands

The Event Response resource manager handles three types of objects: conditions, responses and associations.

Display lscondition, lsresponse, and lscondresp

Creation mkcondition, mkresponse, and mkcondresp

Modification chcondition and chresponse

Removal rmcondition, rmresponse, and rmcondresp

Activation startcondresp and stopcondresp

The monitoring of events using the CLI follows the same principles as when using the GUI:

1. First, define the kind of event you want to monitor.

2. Define the response you want the system to exercise when the events occurs. It is made from either the predefined actions or any actions you provide (programs and scripts).

3. Define associations between the conditions you want to monitor and the responses you have chosen.

4. Activate the monitoring of the actions by starting the associations.


Resources:

A Practical Guide for Resource Monitoring and Control (RMC)


Taking Notes from AIX 5L Administration

Taking Notes from AIX 5L Administration

Linux Affinit

RPM Format

you can install native Linux applications using the familiar rpm commands used with Linux on POWER platforms.

You can also install RPM format Linux programs using the new Install Wizard from IBM, geninstall. With the Install Wizard, you can install AIX installp packages as well as any of the following software package formats:

InstallShield Multi-Platform (ISMP); Red Hat Package Manager (RPM); Uniform Device Interface (UDI)

Hot Spot Management

To find the hot spots, you turn on statistics with the lvmstat command using the -e option.

To turn on statistic gathering: # lvmstat -e -v rootvg

To turn off statistic gathering:# lvmstat -d -v rootvg

View VG statistics by LV: # lvmstat -v rootvg

View LV statistics by LP: # lvmstat -l hd3

lvmstat: Reports input/output statistics for logical partitions, logical volumes and volume groups. Also reports pbuf and blocked I/O statistics and allows pbuf allocation changes to volume groups.

The bootinfo -y command shows if your system is 32-bit or 64-bit enabled..

The bootinfo -K command returns the current kernel mode information for your system directly as an integer value of 32 or 64.

The svmon -P <process_ID> command shows if the currently running process is 32- or 64-bit and whether it is multithreaded. This information is found in the 64-bit andMthrd columns.

Dynamic CPU Deallocation

To enable CPU deallocation, you can use the AIX System Management Interface Tool (SMIT) or the command:

# chdev -l sys0 -a cpuguard='enable'

After making the configuration change, you can verifywith: # lsattr -El sys0

you also have the option to deallocate CPUs on the flyfrom the command line.

This is very useful for scalability testing. A CPU is deallocated using the logical CPU number, where the logical CPU number is an integer between 0 and n–1. The value of n is the total number of CPUs in the system. To deallocate CPU

number 3, use the following command: # cpu_deallocate 3

lsattr: Displays attribute characteristics and possible values of attributes for devices in the system.

Web-Based System Manager (WebSM)

You can also enable WebSM to a client/server environment. In this environment you can manage the system from a remote browser: # /usr/websm/bin/wsmserver -enable

allow the remote system access to your X environment and export the display back to your local machine:

# xhost <remote_machine> or # xhost + to allow all machines to have X access locally.

# export DISPLAY=<local_hostname_or_IP_address>:0.0

man PAGES: # man {command | file | subroutine | system call}

POWER Processors and Systems

LPAR

The LPAR lets you slice and dice your machine into up to 16 logical partitions. Each separate partition has its own set of resources. For example, you can assign processors, memory, and specific I/O ports to belong to a specific LPAR.

System Management of the p690

The pSeries 690 is managed on the IBM Hardware Maintenance Console for pSeries, or HMC. The HMC has a GUI interface system running Linux on an 32-bit Intel platform.

AIX 5L Kernel

Threads Versus Processes

Each thread is a dispatchable unit of work. Multiple threads can be executing simultaneously on multiple CPUs. Threads provide the means of overlapping, multiplexing, and parallelizing operations within a process. The process structure is separated into dispatchable components and global resources. Threads are peer entities and share global

resources such as address space. The nice thing about threads is that they can service multiple requests without the added overhead of creating multiple processes through fork and exec system calls.

a new device can be plugged into a running system and made immediately available for use by updating ODMinformation with the cfgmgr command.

# cfgmgr Update ODM information

System Management Tools

AIX System Management Interface Tool (SMIT), AIX Distributed SMIT (DSMIT), and the AIX Web-Based System

Manager (WebSM).

SMIT

SMIT is the base system administration management tool for AIX, SMIT uses a menu-driven interface that streamlines and simplifies the complexity of many system management activities. SMIT provides both ASCII andMotif-based user interfaces.

When using SMIT, two files are created, the smit.script and the smit.log files. The SMIT script facility may be used to assist in creating complex administration scripts, and the logging facility keeps a log of the tasks that were performed on the system, as well as the date/time the commands were executed.

USING SMI: # smit; # smit user

You can also use the F6 key within SMIT to display the actual AIX command and arguments it will invoke before committing the update.

SMIT Log File

SMIT creates an audit log of each SMIT session in the user’s $HOME directory named smit.log.

The location of the log file may be set using the smit -l <PathName> option. The level of logging verbosity may be increased using smit -vt. # smit -l /tmp/smit.log -vt Use /tmp to hold log file

Special Symbols Used in SMIT Panels

* A value is required for this field.

# A numeric value is required for this field.

X A hexadecimal value is required for this field.

+ A pop-up list is available for this field using F4 (or ESC-4).

[ ] A field can be typed in.

< > Data displayed is larger than the field display. The < and > indicate the overflow direction.

/ A file/directory pathname is required for this field.

? Any value entered will not be displayed.

SMIT Script File

SMIT appends the AIX commands invoked during the session to a local $HOME/smit.script file. The script file information can be used to create complex management scripts or review the commands invoked during a previous

session. Use the smit -s <PathName> option to create a script file in a location other than your home directory.

# smit -s /tmp/smit.script Create a script file in /tmp

SMIT FastPaths

The FastPath name is included as an argument to the smit command to enter a task panel directly.

A set of four prefixes, ls, mk, ch, and rm, are appended to a root task name in the list, make, change, and remove operating system objects. For example, to make a TTY device, the FastPath or command name is mktty.

FastPath and Command Operations and Objects

AIX administration commands are made up using two parts: the operation followed by an operation object:

Operation mk, ls, ch, rm

Objects dev, user, fs, vg, pv, tty, cd, tape, etc.

Putting the operation with the object will produce a direct SMIT FastPath, in most cases.

SMIT FastPath for Application Submenus

Software installation and maintenance install

Software license management licenses

Device management dev

System Storage management (physical and logical storage) storage

Journaled filesystem management jfs

Logical volume management lvm

NFS management nfs

Security management security

User management user or users

Communications applications and services commo

TCP/IP management tcpip

Problem determination problem

System diagnostics diag

Performance and resource scheduling performance

System environments system

Processes and subsystems src

smit date;

SMIT object class names and object identifiers are listed in the SMIT log file when SMIT is invoked with the verbose trace -vt option. # smit -vt SMIT verbose tracing

WEB-BASED SYSTEM MANAGER

AIX 5L leans heavily toward theWebSM, and it appears that some new features may be available only via theWebSM in coming AIX releases. WebSM can run in stand-alone or client/server modes. WebSMcan be started from the AIX Common Desktop Environment (CDE) application manager System_Admin window or from the command line using the wsm command.

AIX Installation and Maintenance

To identify the version and release numbers for an existing system, use the oslevel command. The oslevel command reports the current level of the operating system by using a subset of the filesets installed on the system. The oslevel -q command will display any base migration upgrades that are installed. If oslevel shows only 4.3.2.0 and the -q option shows 4.3.3.0 is present, then oslevel -l 4.3.3.0 will help identify the missing filesets.

The instfix command is

a good tool for ML analysis. For example, the command instfix -i | grep AIX_ML will display all maintenance levels that have been installed on your system. To list only the latest maintenance level, you can use the oslevel -r command.

Another command that is useful for system information is the uname command. Issue the uname -a command to display the UNIX flavor, the system hostname, the operating system release, the operating system version, and the machine ID number. You can also get the version and release of the operating system with the uname -vr command.

Your system maintenance level can be obtained from bos.rte history. Use the lslpp -h command to display the maintenance history and state. The -h option is best for reviewing what older filesets were installed and on what date the installation took place. The -l option will display the current patch level as updated to the /usr/lib/objrepos database

# lslpp -h bos.rte -----Short listing

A quick snapshot of maintenance level and state can be obtained using the -L option to lslpp.

# lslpp -L bos.rte

APPLY and COMMIT

The AIX default is to add new software and maintenance to the system using the COMMIT option as opposed to the APPLY state. The APPLY option keeps a history file and a backup copy of each object replaced by the software update. If you are not happy with the update, you can REJECT the filesets and restore the previous version.

# installp -qa -d /usr/sys/inst.images -X all --- APPLY updates

# installp -rB -X all --- REJECT updates

Once satisfied with the update, you can COMMIT the update to remove the backup copy of the previous version.

# installp -c -g -X all or # smitty install_commit

COMMIT updates

SMIT FastPath

# installp -qa -d /cdrom/usr/sys/inst.images -c -N all

In the event that you must remove a committed lpp, you can invoke the deinstall function of the installp command. This -u option will remove product files, associated maintenance, and vital product data (VPD) from the system regardless of the product installation state.

# installp -u --- fileset_name Remove software

As with all installp command options, it is always a good practice to add the -p option to preview the results first!

# installp -up fileset_name 1>/tmp/output 2>&1

# view /tmp/output

System State

use the stopsrc command to shut down subsystems and subservers.

# stopsrc -g tcpip ---- Stop the TCP/IP subsystem group

When updating your operating system or a group of products, it is easier to reduce system activity by shutting down to maintenance mode. This will stop all applications and subsystems and restrict access to the system other than from the system console. # shutdown -m

If maintenance mode is not required, you can temporarily inhibit nonroot login access by touching a /etc/nologin file. Each time a nonroot user attempts to log in, the contents of the /etc/nologin file are displayed.

run the bootinfo -b command to find the last boot device: bootinfo -b

lspv: Displays information about a physical volume within a volume group.

Lsdev: Displays devices in the system and their characteristics.

lsdev -C -l hdisk0

Use df and lsvg to total the used and free space making up the root file systems and the root volume group.

df -v / /usr /tmp /var; lsvg rootvg

lsvg Displays information about volume groups.

AIX System Boot and Shutdown

If you have applications that you want to always stop before a system shutdown or reboot, put the commands or script name in the /etc/rc.shutdown file.

If you suspect hardware problems or the system won’t boot, use the pSeries or RS/6000 diagnostic programs using the diag command to assist in determining the failure.

To view the special boot log file the system writes all the boot information to, use the alog -t boot -o command.

AIX Runtime Configuration

SYSTEM ENVIRONMENTS

To get to the SMIT catchall runtime setting panel, enter the SMIT FastPath smit system.

Stop the System # smit shutdown

Time Settings # smit chtz_date

Language Environment # smit mlang

Set Characteristics of the Operating System # smit chgsys

Number of Licensed Users # smit chlicense

Manage Floating User Licenses # smit netls_server

Broadcast Message to All Users # smitty wall or wall $message

System Logs # smit logs

Dump Device # smit dump

Internet and Documentation Services # smit web_configure

Change the System User Interface # smit dtconfig

Manage Remote Reboot Facility # smit rrbtty

Manage System Hang Detection # smit shd

Monitor the /var/adm/ras/errlog log file regularly to scan for intermittent system problems using the errpt command..

AIX Device Configuration Manager(cfgmgr) and the Object Data Manager (ODM)

AIX allows dynamic kernel extensions and device driver installation to a running kernel and dynamic device configuration through the object data manager (ODM)

dbm routines reduced large configuration tables into a database of key-content pairs. Items in a database are retrieved in one or two file I/Os, while dbm databases are represented as an index file, *.dir, and a data file, *.pag.

IBM decided to take the doctrine one step further by introducing a hierarchical object-oriented database for configuration data called the object data manager, or ODM. The ODMcentralizes a number of the standard configuration tables into a single management structure with update validation. AIX commands and kernel routines access ODMobjects using SQL-like semantics.

ODM COMPONENTS

The ODM is comprised of a database of object classesWithin each object class is a list of objects. Each object within each object class has a set of object descriptors that describes that particular object

Each component of the ODM—object class, object and object descriptor—has one or more commands that control adding, deleting, changing, or querying the component.

To add or delete object classes, we use the odmcreate and odmdrop commands. The odmdrop command is a very dangerous command! The odmdrop command deletes the entire object class and can leave your system useless.

To add, change, delete, and query an object, within an object class, we use the odmadd, odmchange, odmdelete, and odmget commands. Use odmget to query the database for either predefined or customized objects.

To look at the object descriptors that show individual attributes of the object, use the odmshow command. The odmshow command allows you to look at how the object class is put together.

odmget -q name=hdisk0 CuDv ---- ODM object that describes hdisk0

odmget -q type=osdisk PdDv ---- ODM object that describes the predefined disk type osdisk

odmget -q parent=scsi0 CuDv ---- ODM objects within CuDv attached to parent device scsi0

You can also use the like operator with the odmget command:

odmget -q "name like hdisk?" CuDv --- All ODM customized disk devices

ODM Database

The ODM database manages almost everything on the system. The ODM manages the data associated with all of the devices on the system, some of the software on the system, the System Resource Controller, Error Logging and System Dump Facilities, the SMIT menus, Network Install Manager (NIM), and the configuration of TCP/IP.

Object classes are implemented as standard UNIX files in a directory that represents the ODM database.

The /etc/objrepos directory holds all of the customized devices, or devices that have been found and configured on the system. The /usr/lib/objrepos directory contains all of the pre-defined device object classes, or devices that are supported but not currently installed or active on the system, as well as the LPP updates as they are applied. The /usr/share/lib/objrepos directory holds the share part of each LPP, the four object classes used by the Software Vital Product Data (SWVPD), which are lpp, history, inventory, and product.

The ODMDIR environment variable defines the directory path used as the default ODM database directory. The default ODMDIR path is set to /etc/objrepos by the /etc/environment file.

/etc/objrepos --- Default object class directory

/usr/lib/objrepos --- Additional ODM directory

Objects and Object Classes

ODM objects are the data items that make up the object classes. Object attributes are mapped to a C language structure that represents the object class definition in a stanza format. The object class definition describes the descriptor-value pairs that make up an object. Object classes may be relationally joined to other object classes using a special

link descriptor. Initially, the object class definition is constructed as an ASCII text file identified by the *.cre extension. This description file is read by the odmcreate command to create the object class. The result is an empty object class and an *.h header file that may be used by application programs to populate and manipulate members in the object class.

# odmcreate inventory.cre --- Creates an empty object class called Inventory

The next step is to add some data to the Inventory object class. We again create an ANSI text file to store the entries and then we use the odmadd command to populate the Inventory object class. To populate the Inventory object class with the data stored in the NewInventory file we use the odmadd command.

# odmadd NewInventory

# odmget -q color=black Inventory

odmget -q quantity=20 Inventory > /tmp/Inventory.out

odmdelete -q quantity=20 -o Inventory

To remove an object class, we use the odmdrop command: odmdrop -o Inventory

The object class definition may specify a method descriptor. The method defines a program that is to be invoked by the odm_run_method routine. The method updates the state of the object.

Methods are represented as null-terminated, 255-character strings. The ampersand (&) may be appended to the method for asynchronous execution.

Most Common ODM Commands

# smitty devices SMIT FastPath

# mkdev -l tty0 Add a TTY device

# lsdev -C -s scsi –H List existing SCSI devices

# chdev -l rmt0 -a block_size=0 Change tape block size

# lsattr -D -l rmt0 List tape attributes

# rmdev -l rmt0 Turn off the tape device but keep the definition

# rmdev -l rmt0 –d Remove a tape device

# cfgmgr Update ODM and kernel

DEVICE CONFIGURATION

Device interface definitions and configuration attributes are stored as objects in the ODM database. Each time the system is booted, the cfgmgr command walks the I/O bus and identifies all devices present on the system. Device location and type information is stored in the ODM, and the associated configuration rules and initialization methods are run to make the devices available for use.

Configuration Manager can also be invoked on a running system from the SMIT devices menus or by executing cfgmgr, mkdev, chdev, or rmdev from the command line. This feature allows you to make new devices available without requiring a system reboot.

PREDEFINED AND CUSTOMIZED DEVICES

Device configuration information is separated into predefined and customized object classes. Predefined object class information represents default configuration information for all devices supported by AIX. Customized object classes represent the devices actually present on the system.

/etc/objrepos/Pdxxx ODM predefined Attributes, Connections, and Devices:

PdAt PdCn PdDv

/etc/objrepos/Cuxxx ODM customized Attributes, Dependencies, Device

Drivers and Vital Product Data:

CuAt CuDep CuDv CuDvDr CuVPD Config_Rules CuOD

/usr/share/lib/objrepos Components of the SoftWare Vital Product

Data (SWVPD)

lpp history inventory product

New ODM Class for AIX 5L (CuOD)

CuOD, or Customized On-Demand object class

The CuDD class allows you to order a new pSeries machine with extra CPU capacity, and you can dynamically allocate

these new CPUs on-the-fly.

Logical Partitioning (LPAR) starting with AIX 5L, version 5.2, allows you to create 1–16 logical systems within a single machine. These logical system partitions are considered independent systems to the outside world and can be managed as independent systems. IBM allows you to create these LPARs by specifying the exact number of processors and memory for the logical system, and allows different OS (AIX and Linux!) versions to run within the same machine.

You can display object class definitions using the odmshow command.

# odmshow <ObjectClassName>

SMALL COMPUTER SYSTEM INTERFACE

The SCSI standard defines a generic interface and command protocol that will support most device types. Devices are attached in a daisy-chain fashion to the host adapter. The total chain length cannot exceed the distance maximum for the adapter type.

SCSI Over IP (iSCSI)

SCSI over IP, or iSCSI, has emerged as the standard for Networked Attached Storage (NAS). The most common use of NAS is for file sharing. The iSCSI standard allows SCSI devices to communicate on the IP network. Normally, iSCSI is

implemented over a gigabit IP network.

The AIX Logical Volume Manager (LVM)

RAID

A Redundant Array of Independent Disks (RAID) brings additional fault tolerance into storage support through the use of data mirroring and striping technologies. In addition to data fault tolerance and availability, RAID can maximize read I/O performance by spreading access across multiple disk devices, which allows more spindles to get into action to read and write data.

RAID level 0 is used to stripe data across disks without any parity recovery support. RAID level 1 is what we have come to know as mirroring.

Using RAID 10 increases performance and reliability by taking the RAID level 0 stripped disks and mirroring them with RAID level 1 technology.

RAID Levels

RAID 0 Striped data, no parity

RAID 1 Mirrored data

RAID 2 Striped data with parity

RAID 3 Striped data with parity drive

RAID 4 Striped data with parity and asynchronous read access

RAID 5 Striped data with striped parity

DISK INSTALLATION

You can update device information online by invoking cfgmgr or using the mkdev command. The new disk is assigned the next available hdisk<nn> label.

# mkdev -c disk -s scsi -t osdisk -p scsi2 -w 8,0 -l hdisk10 -a pv=yes

# cfgmgr -vl scsi2 (or) # cfgmgr -v (or) # smitty cfgmgr

# lspv; # lsdev -Cc disk

Use the lspv and lsdev commands shown here to make sure that the disk has a PVID and is in an available state.

An Easier Way to Rename Disks

The first step is to list the disk locations within the system: # lsdev -Cc disk

Next we need to list the adapter locations within the system: # lsdev -Cc adapter | grep scsi

We now need a list of the volume groups involved in this disk reassignment: # lspv

Before we make any changes, we first need to make the target volume group unavailable: # varyoffvg appvg

we can remove the target devices on scsi1: for N in 2 3 4 do rmdev -dl hdisk$N done

Preview the commands to add the devices with new names:

for NA in 10@8,0 12@9,0 14@10,0

do

N=$(echo $NA | awk -F@ '{print $1}')

A=$(echo $NA | awk -F@ '{print $2}')

echo "mkdev -c disk -s scsi -t osdisk -p scsi1 -w $A -l hdisk$N -a pv=yes"

done

The preview output will show these commands:

mkdev -c disk -s scsi -t osdisk -p scsi1 -w 8,0 -l hdisk10 -a pv=yes

mkdev -c disk -s scsi -t osdisk -p scsi1 -w 9,0 -l hdisk12 -a pv=yes

mkdev -c disk -s scsi -t osdisk -p scsi1 -w 10,0 -l hdisk14 -a pv=yes

Now remove the echo around themkdev command to actually execute the commands.

List the new device names: lsdev -Cc disk

List the hdisks numbers, PVIDs, and the associated volume groups: lspv

lqueryvg -Atp hdisk10

Make the volume group available again by varying appvg back online: varyonvg appvg No need for importvg in this case.

Check for the volume groups associated with the disks: lspv

The chpv command has a -h flag the sets the characteristics and the allocation permissions of the physical volume so that it can be used as a hot spare. The -h flag has no effect for nonmirrored logical volumes. The -h flag can take either y or n as a parameter to add or remove the disk to or from the hot spare pool.

Set up the hot spare disks: # chpv -h 'y' hdisk3 hdisk4 # -h [y|n]

Just like the chpv command, the chvg command uses the -h flag to set the sparing characteristics for the volume group, which is specified by the VolumeGroup parameter.

Next we will enable a hot spare for the volume group: chvg -h'y' -s'y' appvg

check for errors: # errpt | head -3

check the daemon:# lssrc -s ctrmc

check the process tree:# pstree -p 13934 or just use the AIX ps command: # ps -ef | grep rmcd

INTRODUCING THE LVM

We start with the largest structure, which is one or more disks grouped together to make a volume group (VG). Within the VG, we create logical volumes (LVs). The LV is made up of logical partitions (LPs) that are mapped to actual disk partitions called physical partitions (PPs). On top of the LV, we can create filesystems, which have a mount point or directory. We can also have raw logical volumes, which are nothing more than an LV without a mount point that is accessed through its raw device name. Databases like to use raw LVs, but it is a big no-no to pair raw LVs with High Availability Cluster Multi-Processing (HACMP)!

The LVM consists of the following components:

Volume groups (Vgs); • Physical volumes (Pvs); • Logical volumes (Lvs); • Logical partitions (Lps);• Physical partitions (PPs)

A physical partition (PP) is a section of disk space that can be used for logical volumes (LVs). Typically, logical volumes are for filesystems and for paging space. Traditional legacy UNIX systems restrict partitions to contiguous space on a single disk such that a single filesystem cannot be any larger than the size of available contiguous space on a

single disk. However, AIX uses the concept of logical volumes (LVs) to map to physical disk space. A logical volume is represented by a mapping of logical partitions (LPs) to physical partitions (PPs) residing on one or more physical disks within the same volume group (VG). One to three physical partitions may be mapped to a logical partition, which is called mirroring. Logical volumes are allocated from logical partitions within a single volume group (VG).

The LVM Components

The LVM components are described in the following five items.

1. A physical volume (PV) is a single piece of disk storage.

2. A volume group (VG) is a collection of one or more physical volumes (PVs).

A VG is the largest unit of storage that AIX can allocate. Within the VG, we create smaller structures like logical volumes (LVs) and filesystems. A nice thing about a VG is that it can be moved from one machine to another machine and imported to be a resident VG. Since a VG is made up of PVs, we break up a VG into physical partitions (PPs), which are the smallest unit of measurement within a VG.

3. A physical partition (PP) is a small block of a PV, or physical volume. A PP is the basic building block unit of storage. So a PV is split up into equal-sized blocks of storage called PPs, or physical partitions. The default PP size is 4MB. All PPs within a VG are the same size, and you cannot change the PP size dynamically.

4. A logical partition (LP) is a map to a physical partition (PP) on a disk drive. The LPs are allocated in contiguous units. However, the PP that each LP maps to can be any PP within the same VG. This is the mechanism

that allows a filesystem to span multiple disks in AIX.

5. A logical volume (LV) is a collection of one or more logical partitions (LPs) within the same volume group (VG). The PPs that each LP is mapped to can be any PP within the same VG. This LP mapping allows the disk space to be presented to the system as contiguous disk space when in reality it is the LP map that is contiguous. On top of these logical volumes, we can build filesystems that can be accessed through a mount point, or directory. Each logicalpartition (LP) that makes up a logical volume is the same size as the PP size (4MB by default); they are one and the same, with each LP mapping to a PP.

Striping is a mechanism that allows you to distribute a file or filesystem’s data blocks evenly across multiple disks to improve I/O performance.

run a migratepv command to move the data to the new area.

Hot Spot Management

AIX 5L has provided two new commands, lvmstat and migratelp, that enable you to identify and resolve hot spot problems in each logical volume.

Gathering Statistics

# lvmstat -e -v rootvg

The previous command will enable statistic gathering for the rootvg volume group. You can also enable gathering of statistics on a single logical volume by using the -l flag and specifying the logical volume name: # lvmstat -e -l hd2

The lvmstat command will produce reports at the logical volume level. The first report produced by the lvmstat command shows statistics since the last system reboot. The succeeding reports show the statistics since the previous report: lvmstat -v rootvg

The lvmstat also allows reporting over an interval, specified in seconds. To display the top ten logical partitions for the hd2 logical volume: lvmstat -l hd2 -c 10

To display reports continuously every five seconds for the hd2 logical volume: # lvmstat -l hd2 5

To display ten reports every five seconds: # lvmstat -l hd2 5 10

To reset the all of the statistics counters for the rootvg volume group: # lvmstat -v rootvg -C

To disable statistics gathering for the rootvg volume group: # lvmstat -d rootvg

Migrating Physical Partitions

migratelp LVname/LPartnumber[/Copynumber] DestPV[/PPartNumber]

1. To move the first logical partitions of logical volume lv00 to hdisk1, type # migratelp lv00/1 hdisk1

2. To move second mirror copy of the third logical partitions of logical volume hd2 to hdisk5, type:# migratelp hd2/3/2 hdisk5

3. To move third mirror copy of the twenty-fifth logical partitions of logical volume testlv to hundredth partition of hdisk7, type # migratelp testlv/25/3 hdisk7/100

CONFIGURING VOLUME GROUPS

In order for a new disk to be made available to the LVM, the disk must be designated a physical volume and assigned a physical volume identifier (PVID). You create a PVID on a new disk: # chdev -l hdisk<n> -a pv=yes

You can list physical disks on your system using lsdev: # lsdev -C -c disk

To list the PVIDs associated with these disks, use the lspv command: # lspv

To add the new disk to a new or existing volume group, use SMIT or the mkvg and extendvg commands:

# mkvg -f -y vg10 hdisk10 hdisk11

# extendvg -f rootvg hdisk8

# smit mkvg

Create a volume group vg10 using hdisk10 and hdisk11 Add hdisk8 to the rootvg volume group

A volume group identifier (VGID) is assigned to each volume group.

Each VGID in the system is represented by an entry in the /etc/vg directory.

To display the configuration of the existing volume groups on your system, use the lsvg command:

# lsvg -l rootvg ---List the contents of the rootvg volume group

# lsvg -p rootvg ---List physical volumes in the root volume group

Quorum

# varyonvg <VGname> ---Vary a volume group online

# varyonvg -f <VGname> ---Force vary on a volume group

You can take a volume group offline using the varyoffvg command or via SMIT.

# varyoffvg <Vgname>; # smit varyoffvg

To remove or replace physical volumes in a volume group for maintenance purposes:

# reducevg <VGname> <PVname> ---Remove the disk from VG

# rmdev -dl hdisk<n> ---Remove the hdisk definition

Replace the disk...

# cfgmgr -v ---Let the system discover the disk

# lspv --- Find the newly found disk and write down the hdisk #

# chdev -l hdisk<n> -a pv=yes ---Create a PVID for the new hdisk

extendvg <VGname> <hdisk<n> ---Add the new disk into the volume group

An entire volume group may be moved as a unit from one system to another.

# exportvg <Vgname>; # importvg <VGname> <PVname>

Root Volume Group—rootvg

A special VG called rootvg is used by AIX for the operating system’s root filesystems and the default paging areas. It’s a good idea to use separate volume groups for user and local application filesystems.

Adding a Volume Group

To specify a big volume group, add the -B switch to the mkvg command. To create a normal volume group called appvg using hdisk4 that has 16MB PP size and can have up to 16 disks: # mkvg -s 16 -t2 -y appvg hdisk4

The -s 16 specifies a 16MB PP size, and the -t2 specifies a t-factor of 2. The -y switch specifies the name of the volume group, appvg.

To create a big volume group called bigappvg that can have up to 16 disks and 8128 partitions per disk and that uses hdisk5 through hdisk10: # mkvg -B -t8 -y bigappvg hdisk5 hdisk6 hdisk7 hdisk8 hdisk9 hdisk10

We can also change our normal appvg volume group to be a big volume group: # chvg -B appvg

As an example, we want to use our bigappvg volume group on another machine.

On machine A: # exportvg bigappvg

On machine B:# importvg -y bigappvg hdisk5

To find the next available major number on a system:

# lvlstmajor To specify a major number, we use the -V switch. # importvg -V 27 -y bigappvg hdisk5

Varying On and Varying Off Volume Groups

Before a volume group can be accessed, it must be varied on. Also, before you can export a volume group, it must be varied off: # varyoffvg bigappvg; # varyonvg bigappvg

HACMP is an acronym for High Availability Cluster Multi-Processing. HACMP is used to make your application

servers fault resilient by failing over to a standby machine if the primary machine fails.

Adding a Logical Volume: # smitty mklv

Logical Volume Maps

To look at how the LVs are mapped to the PPs, you can use the lslv -m <LVname>.

Raw Logical Volumes Database applications are notorious for requiring raw logical volumes. A raw logical volume is a logical volume without a filesystem. Paging space is also a raw logical volume. The access to raw logical volumes is done by way of the raw device in /dev/r*.

Increasing the Size of a Logical Volume

You can increase the size of a logical volume on-the-fly in AIX using the extendlv command or by the SMIT FastPath smitty extendlv: # extendlv hd9var 5

FILESYSTEMS (JFS AND JFS2)

Information for a particular inode can be displayed using the istat command: # istat 25 /dev/hd6

To find the i node for a particular file, use istat filename.

To find the filename associated with the inode number, use the -inum flag with the find command:

# find /home -xdev -inum 25 -print

Journaled Filesystem Configuration

The native filesystem in AIX is a log-based filesystem called the Journaled Filesystem (JFS). Log-based filesystems like JFS improve recovery by maintaining a circular update log. In the event of a failure, the JFS log is replayed to recover the filesystem state. Log recovery of a filesystem is completed hours faster than a full fsck walk of a filesystem. AIX provides the fsck to assist in disaster recovery; Create or update a JFS filesystem using the SMIT fast path or the crfs and chfs.

The filesystem attributes are recorded in the ODM custom databases and the /etc/filesystems file.

You may remove a filesystem using the rmfs command: # rmfs /n5 or SMIT:# smitty rmjfs

Large Filesystems (JFS-2)

Mounting and Unmounting Filesystems

# mount /dev/hd5 /n5; # umount /n6 or by using SMIT: # smit mountfs

You cannot unmount a filesystem that is busy. Use tools like fuser and the public domain lsof commands to identify which processes and users have files open: # fuser -u /dev/hd1; # lsof /dev/hd1

Filesystems may also be identified as a group by adding a type=<name> parameter for the group in /etc/filesystems. This is specified by the chfs -u groupname flesystem. The group name can then be used to mount or unmount all the filesystems in the group from a single command: # mount -t nfs

To display the currently mounted filesystems and their state, use the df and mount commands:

If you need to immediately unmount a filesystem and it is in use, you can use the fuser command with the -k switch option, for kill. # fuser -k /data01; # umount /data01

Increasing the Size of a Filesystem

Filesystems can be increased on the fly with the SMIT FastPath smitty chfs or the chfs command.

# chfs -a size=40960 /home

To add 40MB to the /home filesystem, issue the following command: # chfs -a size=+81920 /home

Decreasing the Size of a Filesystem

To decrease the size of a filesystem requires doing things the old-fashioned way: back up the filesystem, re-create it, and restore the backup.

PAGING SPACE

Paging (swap) space is used by AIX to support virtual memory services. When free memory becomes low or exhausted, AIX moves data pages from primary memory storage to the paging areas according to a least recently used algorithm. A threshold is maintained for virtual memory usage by the operating system. When the threshold is exceeded, a SIGDANGER signal is sent to all processes. If a second threshold, called the kill level, is exceeded, then the operating system sends SIGKILL signals to the biggest memory offenders. This process continues until memory utilization falls below the threshold levels.

Adding a Paging Space

There are two commands that will display the amount of real memory in your system. Both commands report resident memory in kilobytes: lsattr -El sys0 | grep realmem; bootinfo -r

You can monitor each paging space with the lsps -a command, and you can get a summary of all paging spaces put together using the lsps -s command. The default system paging logical volume is hd6. Each additional paging space logical volume will be assigned paging00, paging01, and so on

Always place paging space logical volumes on disks with the lowest disk activity

To add paging space to the system, you can use smitty mkps SMIT FastPath or the mkps command. You can also change the current paging space allocations using smitty chps SMIT FastPath or the chps command. Paging space is allocated in logical partitions (LPs).

The following command will create a paging space on hdisk1 that is 8 LPs in the rootvg volume group and will be activated immediately and on all subsequent system restarts: mkps -a -n -s8 rootvg hdisk1

Activating and Deactivating a Paging Space

For a paging space to be used, it must be activated. All paging spaces are defined in the /etc/swapspaces file. You can activate the paging spaces dynamically by using the swapon command: swapon /dev/paging03

To activate all paging spaces at once: swapon -a

New to AIX 5L is the swapoff command. The swapoff command will enable you to deactivate a paging space on the fly: swapoff /dev/paging03

Removing an Active Paging Space

Before you can remove a paging space from the system, the paging space must be deactivated,

swapoff /dev/paging03; rmps paging03

Reducing the Size of an Active Paging Space

IBM added another switch to the chps command, -d. This command will execute the shrinkps shell script that will jump through all of the extra hoops for you.

To reduce the size of the paging03 paging space by two logical partitions: chps -d2 paging03

Handy Commands for Paging Space

mkps Adds an additional paging space

rmps Removes an inactive paging space

chps Changes the attributes of a paging space

lsps Displays paging space statistics

swapoff Deactivates paging space on a running system

swapon Activates paging space

chps –s Increases the size of a paging space

chps –d Dynamically decreases the size of a paging space on a running system shrinkps Shell script that chps -d executes to shrink paging space svmon Takes a snapshot of virtual memory usage. Part of the perfagent.tools fileset

topas IBM’s top/monitor clone vmstat Reports virtual memory statistics sar System Activity Report istat Status of inodes slibclean Removes any currently unused modules in kernel or library memory

Moving Logical Volumes Within the Same Volume Group: migratepv or smitty migratepv

Use lslv to identify the current logical volume location and produce a physical disk map.

Use the migratepv -l command to move the logical volume to its new location.

For example, to move the paging03 paging space from hdisk1 to hdisk2 in rootvg: # migratepv -l paging03 hdisk1 hdisk2

Moving Logical Volumes Between Volume Groups

To move a logical volume to a different volume group, we use the copy LV command, cplv. For example, to move the dat01 LV from rootvg to appvg, issue the following three commands:

# cplv -v appvg -y new01 dat01; # chlv -n old01 dat01; # chlv -n dat01 new01

Moving Volume Groups

You can also move a volume group from one system to another using the exportvg and importvg commands or the SMIT vg FastPath.

Check error log with the errpt command.

If a filesystem has become corrupted, take the filesystem offline and run the fsck command, which will walk the filesystem structures and identify problem entries or chains. Occasionally a volume group–related ODM update may abort, leaving the volume group locked. To unlock a volume group, issue a getlvodm/putlvodm sequence using the volume group name as an argument: # chvg -u rootvg

Network Configuration and Customization

TCP/IP

NETWORK DEVICES AND INTERFACES

To display existing network interfaces in an AIX system: lsdev -Cc if

Ethernet It is a broadcast-based protocol that uses collision detection and avoidance to regulate network traffic. The Carrier Sense Multiple Access/Collision Detection (CSMA/CD) protocol allows any system to start transmission when the wire is clear, but requires it to back off and wait a pseudorandom period of time if it detects that it has a collision. If it detects a collision again after retransmission, it will wait for a random period of time before trying again.

Configure your Ethernet adapter via the SMIT FastPath smit chgenet. You may want to increase the value of the TRANSMIT buffer. The value represents a queue of buffers for outgoing packets. Values may range from 20 to 150.

# smit chgenet

ADDRESSING

IPv4 Addressing

The address is split into two, representing a network address and a host address on that network.

Netmasks and Subnetting

Subnetting allows an organization to subdivide the address range it has been allocated.

A netmask is most commonly written as a dotted decimal address such as 255.255.0.0. For each bit in the netmask that is set to 1, the corresponding bit in the address is part of the network address. For each bit in the netmask that is set to 0, the corresponding bit in the address is part of the host address.

A new terminology called Classless Inter-Domain Routing (CIDR) has expanded the concept of simple class A, B, and C networks. CIDR defines integer prefixes from /1 through /30 that can be used to allocate a block of addresses of whatever size is needed rather than always allocating addresses along class A, B, or C boundaries.

A common plan is to configure public IP addresses on gateways or firewalls that will provide Network Address Translation (NAT) for your private internal IP networks. Three ranges of private IPv4 addresses are reserved, by the Internet Assigned Numbers Authority (IANA), for use by all enterprises for their internal network administration.

This private IP space provides a safeguard to protect the internal network while conserving addresses in the public IP space of the Internet.

Private IP Range CIDR Addresses

10.0.0.010–255.255.255 10/8 16,000,000

172.16.0.0–172.31.255.255 172.16/12 1,024,000

192.168.0.0–192.168.255.255 192.168/16 64,000

IPv6 Addressing

The IPv6 address is a 128-bit number represented as eight 16-bit hexadecimal fields separated by colons. IPv6 also defines two types of addresses that are used for compatibility with Ipv4. The first type has a prefix of 0:0:0:0:0:0:ffff/96. The final 32 bits of the address will be an IPv4 address. This type of address is used simply to represent an IPv4 address in Ipv6 format and is called an IPv4 mapped address. The other type of address has a prefix of 0:0:0:0:0:0:0/96. The final 32 bits will again be an IPv4 address. This is called an Ipv4-compatible address and is used for IPv6 automatic tunneling. Automatic tunneling is used for IPv6 nodes that must send to other IPv6 nodes across an IPv4 network. If both systems have IPv4-compatible addresses, the sender can send the packet to the

receiver's IPv4-compatible address and the packet will be automatically encapsulated within an IPv4 packet so that it can be sent across the IPv4 network.

NDP Neighbor Discovery Protocol (NDP) is the IPv6 replacement for ARP

Aliasing: smit inetalias

VIPA

A virtual IP address (VIPA) is an address that is not associated with a particular physical network adapter. VIPAs can be used in situations where network communications might need to continue even in the event of a network interface going down. AIX 5L allows an administrator to configure an interface called vi0 and assign an address to it. This address will be used as the default source address for any packets sent unless the application is explicitly bound to a different address

Kernel Routing Table

The routing table can be displayed in AIX 5L with the netstat -r command.

Static Routing

The Systems Administrator can add routes to the kernel routing tables using route, chdev, or SMIT FastPath smit mkroute.

DOMAIN NAME SYSTEM

For a small network, cross-references between IP address and symbolic names are kept in an /etc/hosts file on each machine on the network. For large networks, TCP/IP applications use the Domain Name System (DNS), which is

a client/server application with a distributed database for mapping IP addresses to names.

Name Servers

AIX 5L ships both compiled versions of Berkeley Internet Name Domain (BIND), version 4 and version 8.

Host Names: hostname, smit mkhostname

Naming Resolution Process

Applications resolve names and IP addresses via two library routines, gethostbyname() and gethostbyaddress(). These routines are part of a name translation library, called resolver.

/etc/resolv.conf, /etc/hosts

TCP/IP SUBSYSTEMS

TCP/IP daemons run as subsystems under AIX. they are under the control of the system resource controller (SRC).

Subservers are enabled in the /etc/inetd.conf file and started as needed by inetd. All the subsystems associated with the TCP/IP group should be started from the /etc/rc.tcpip script. The start() function in this script will use the startsrc –s command to start each subsystem that has been uncommented.

# lssrc –T ----List all the inetd subservers

# lssrc –t ftp ----List the status of the ftp subserver

# lssrc –g tcpip ----List all the TCP/IP subsystems

# stopsrc –g tcpip ----Stop all the TCP/IP subsystems

# startsrc –s inetd ----Start the subsystems one by one

# refresh –s inetd ----Makes the daemon read configuration files

# smit onfigtcpc

Master Daemon inetd

Configuration for inetd is located in the /etc/inetd.conf and /etc/services files. Entries in the /etc/inetd.conf file indicate the service name and startup information. The /etc/service file lists the service name, whether it uses TCP and/ or UDP protocols, and the well-known port number associated with the service.

Any time updates are made to either of these files; you will need to refresh inetd. This can be done with the SRC refresh command or by sending a hangup signal to the inetd process.

# refresh inetd; # kill -HUP <inetd-pid>

Other Network Daemons

Not all TCP/IP server daemons run under the control of inetd. You may also choose to run an inetd subserver as a stand-alone daemon to service high-traffic loads. This eliminates the overhead involved in restarting the daemon for each service request. Start stand-alone service daemons using an entry in /etc/inittab or from a local /etc/rc.tcpip file.

Startup Configuration: /etc/inittab, /etc/rc.net and /etc/rc.tcpip

DHCP: dhcpcd - client daemon and dhcpsd - server daemon.

TCP/IP CONFIGURATION

Configure TCP/IP: smit tcpip, smit chinet

# ifconfig en0 up

# ifconfig en0 down

# ifconfig en0 detach ---Detach en0 before making changes to ent0

# ifconfig en0 123.145.100.18 netmask 0xffffff00 broadcast

arp Command

The arp command can be used to view and configure the ARP table in AIX 5L: arp -a;

autoconf6 Command

IPv6 requires that each network interface have a link local address that can be used to communicate with routers and other hosts on the local network link. autoconf6 automatically generates and assigns a link local address on each interface that is already configured with an IPv4 address. In addition, it will create the sit0 interface, which is used for automatic tunneling. The sit0 interface will be created using the IPv4 address of the first configured IPv4 interface, unless a different interface is specified using the -m flag of autoconf6.

Routing Configuration: route; smit route

The route Command

This command adds a route to network 10.1.1.0 through gateway 10.2.0.1. To delete the route, the subcommand add would be replaced with delete. To add and delete Ipv6 routes using the route command, the flag -inet6 must be used.

Network Options

Network options are used to tune various behaviors of TCP/IP on AIX 5L. They are set and displayed using the Network Options command, no.

# no –a ----Display all network options and their current values

# no -o <option name> ----Display a single network option and its current value

# no -o <option name>=<value> ----Set the value of a network option

# no -d <option name> ----Set a network option to its default value

Using $HOME/.rhosts and /etc/hosts.equiv files allows use of the Berkeley r commands without requiring a password.

TCP/IP TROUBLESHOOTING

The most important of these are probably the packet tracing tools, iptrace and tcpdump.

traceroute allows the administrator to see which router is the last to receive the packets.

The simple ping command can also be used to debug network problems.

The netstat command can be used to display interface statistics and connection status for a given workstation or host. The -a option will show all server connections on the system, including the port numbers.

Distributed File System

Distributed Computing Environment (DCE) Distributed File System (DFS)

Filesystem Structure The DFS shared root is named /..., and cell subdirectories follow as /.../cellname/. The shorthand strings “/.:” and “/:” indicate the local cell and root fileset respectively. Clients need only mount root “/...”–level DFS space to access all filesets exported by DFS file servers in the cell space.

AIX DFS

/ home/username/.kshrc/... /cellname /fs/ home/username/.kshrc

/ .: /fs /home/username/.kshrc

/ : / home /username/.kshrc

DFS uses an object as the mount interface. To mount a DFS filesystem, use the fts crmount command. # fts crmount -dir [PathName] -fileset [FilesetName | FilesetID]

Use the fts lsmount command to see if a directory serves as a mount point for a fileset.

# fts lsmount -dir /.:/fs/home

Linux Affinity with AIX

AIX Toolbox for Linux Applications

http://www.ibm.com/servers/aix/products/aixos/linux/

If you want to change your desktop to either GNOME or KDE, make sure that three additional AIX filesets are installed: X11.adt.lib, X11.apps.xdm, and X11.samples.apps.clients.

# lslpp -L X11.adt.lib X11.apps.xdm X11.samples.apps.clients

Red Hat Package Manager (RPM)

Before installing the whole AIX Toolbox for Linux, you must first install the Red Hat Package Manager (RPM). This fileset must be installed as a traditional AIX fileset using installp, SMIT, or WebSM.

# cd /cdrom/INSTALLP/ppc

# installp –qacXg -d . rpm.rte or use the SMIT FastPath smitty install_latest

# rpm -qa

# rpm -hiv */*

# rpm -hiv apache*

# rpm -hiv emacs* samba*

# rpm -hiv gzip-1.2.4a-5.aix4.3.ppc.rpm

Process Management

Process ID 1, init, is the great-grandparent to which all process generations trace their origins. Like a loving grandparent, init takes in the orphan processes that have lost their parents.

You can set the maximum number of processes per user by using the smitty system.

ps auxw; ps -elk

Resource Utilization and Priority

New processes inherit the nice value of their parents. The nice value may be altered dynamically during the process lifetime. The owning UID for a process can lower the process nice value. Only the superuser can improve nice priority.

The nice value can be set from the command line using the nice command when starting a process.

# nice -n <value> <command>

Process owners and the superuser can modify existing process nice values by using the renice command.

# renice <value> -p <PID>

Listing the Top CPU Hogs: topas

alias topprocs="ps aux | tail +2 | sort -k 1.15,1.19nr | head -n 15"

alias top15="ps -ef | grep -v STIME | sort +3 -r | head -n 15"

To view a list of the top 15 most CPU-intensive processes (in descending order) that are not owned by you user ID:

# ps -ef | egrep -v "STIME|$LOGNAME" | sort +3 -r | head -n 15

Binding Kernel Threads to a Processor

The bindprocessor command enables you to bind and unbind kernel threads to or from a particular processor or lists all of the available processors. You need to understand that the actual process is not bound, but the kernel threads are bound.

-q Displays all available processors; • -u Unbinds the threads of the specified process

# bindprocessor 12345 2; # bindprocessor -u 12345

To display the processes and the CPU each binds (BND) to (if any): # ps -e -F comm,pid,user,thcount,bnd | more

PARENT CHILD INHERITANCE

A parent process creates a new child process by invoking the fork() system call. The kernel reserves a vacant PID for the child and copies the attribute data associated with the parent into the child’s proc structure. The child is a clone of the parent until either the child, the parent, or a privileged authority modifies the child’s attributes by way of a system call. The most common method for modifying a child’s proc attributes is by invoking a new program using the exec() system call.

CONTROLLING PROCESSES

# kill [-Signal] [PID PID PID …]

If the PID value is zero (0), the signal is sent to all the processes that have the same process group ID (PGID) of the sender. If the PID value is negative one (–1), the signal is sent to all the processes with the effective user ID(EUID) of the sender. No signal will be sent to the processes with the ID of zero or one.

PID > 0 Sends signal to specified PID integer

PID = 0 Sends signal to PIDs that have PGID equal to the sender

PID = –1 Sends signal to all PIDs with EUID equal to the sender

PID < –1 Sends signal to all PIDs with a PGID equal to the absolute integer specified

To display the set of supported signals, use the -l argument of the kill command: # /bin/kill -l

If you want to send a signal to all your processes except the sending process, use the killall command. Of course, if you are logged in as root, then the killall command will kill all processes except for the very basic processes that are needed to keep the system running. # killall [-signal]

You can also write into your shell scripts the capability to trap exit signals. You can trap any signal except for kill -9.

Rules of Thumb

kill -1 <PID> First try HUP

kill -6 <PID> Then try ABRT

kill -9 <PID> KILL if all else fails

Ignoring hangup

A common problem is starting a command in the background or as a daemon from the command line of a login shell only to find that the command exits when you log out. This is because a hangup (HUP) signal is sent to the process when your terminal connection has been broken. You can specify that these commands are to ignore HUP by using the nohup command.

nohup <command> & Background process ignoring hangup

THE /proc FILESYSTEM

/etc/vfs;lsvfs procfs;lsvfs;ls -l /proc/19898

The object directory contains files that correspond to the files mapped in the address space of the process.

# ls -l /proc/19898/object

The lwp directory contains subdirectories for each kernel thread running in the process. IBM uses the term “lwp” to mean a lightweight process.

# ps -mo THREAD -p 19898

We use the od (octal dump) command to show the contents of the /proc/19898/cred.

SCHEDULED PROCESSES (CRON)

cron logs errors to a log file, /var/adm/cron/log.

crontab

To create a crontab, use your favorite editor to create a table with the following format:

minutes hours day month weekday command

Each of the time-associated fields may be represented as a comma-separated list, a hyphen (-) separated range, or an asterisk (*), which may be used to represent all possible times.

0,30 * * * * /bin/uptime > /dev/console 2>&1

Once you have your crontab file tailored to your liking, hand it off to the cron daemon by invoking the crontab command.

# crontab <YourCrontabFile>

# crontab –e # As the cron table owner

All crontabs are stored in the /var/spool/cron/crontabs directory under the owning user’s name.

/usr/adm/cron/{cron.allow,cron.deny}

/var/adm/cron/cron.deny takes precedent over the cron.allow file.

Ad Hoc Jobs: at, batch

Execute at, specifying the time and the input stream of commands. The job stream is copied to the /usr/spool/cron/atjobs

directory. When the time comes, cron executes the job stream at the specified time. Authorization to run jobs with at is controlled, as in the case of crontab, by listing user names in the /usr/adm/cron/{at.allow,at.deny} files.

# at <time> input <Ctrl-D> Start a job at <time>

Reboot the system at 11:00 PM on the next Saturday. Do not forget the echo or the system will reboot now from the current shell # echo "shutdown -Fr" | at 11 pm sat

List scheduled jobs: # atq <username>

Remove a job:# at -r jobnumber

SYSTEM RESOURCE CONTROLLER

AIX provides a mechanism for controlling and managing sets of programs that function collectively as a unit. This mechanism is called the System Resource Controller (SRC). SRC provides simple command interfaces to display the status of, refresh, start, and stop system services as a single entity.

SRC allows the operator or administrator to operate on a service at the subserver, subsystem, or subsystem

group level.

SRC components

SRC is provided by the srcmstr daemon. The srcmstr daemon is started at boot time by an entry in the /etc/inittab file.

srcmstr:23456789:respawn:/usr/sbin/srcmstr # System resource controller

Subsystems and subserver

configuration information are managed through the use of the {mk,ch,rm}ssys and {mk,ch,rm}server commands.

startsrc Start a subsystem

stopsrc Stop a subsystem

refresh Restart or refresh a subsystem

trace -a, trcon, trcoff, and trcstop Turn on/off tracing of a subsystem

lssrc Display subsystem status

# startsrc -g tcpip Start the TCPIP subsystem group

# refresh -g nfs Refresh the NFS subsystem group

# stopsrc -s qdaemon Stop the qdaemon subsystem

subsystem control may also be invoked using the smitty subsys and smitty subserver

# lssrc -a

System Accounting

Connect Time

Connect-time data is accumulated in the /var/adm/wtmp and /etc/utmp files.

# last -f /var/adm/wtmp | more

Process Resource Usage

Resource utilization information for each process run by the operating system is recorded in the /var/adm/pacct file at process exit.

Disk Usage

You can periodically collect disk usage information for the system and store it in the /var/adm/dtmp file

ACCOUNTING CONFIGURATION

# lslpp -l bos.acct

List of Accounting Records

Filename Data Collected

pacct Active process data

Spacct.<mmdd> Daily active process data (runacct)

qacct Print usage data

utmp Connect session data

dtmp Disk usage data

wtmp Active process data

Setting Up Collection Files

Touching the filename or running nulladm can create a missing file stub.

The nulladm command creates or overwrites the arguments supplied as zero-length files and sets the correct permissions.

# touch /var/adm/pacct

# chown adm /var/adm/pacct

# chgrp adm /var/adm/pacct

# chmod 644 /var/adm/pacct

or # /usr/sbin/acct/nulladm /var/adm/pacct

Only changes the time stamp if the file exists, this will overwrite the file

Identifying Shifts

Configure the /etc/acct/holidays file to reflect your prime-time shift and scheduled holidays

Disk Accounting (/etc/filesystems) Use the chfs command to set the account stanza to true for each file system you intend to monitor.

The lsfs command lists the characteristics of each filesystem, and the last column, which usually word-wraps on a standard terminal, is the account stanza.

# lsfs; # chfs -a account=true /home; # lsfs /home

Report Directories

Manually create the report subdirectories nite, fiscal, and sum with adm ownership in the /var/adm/acct directory.

# su – ad; # cd /var/adm/acct; # mkdir nite fiscal sum

Performance Tools

The vmstat command reports information about kernel threads, memory, disk I/O, traps, and processor activity.

vmstat -It 5 2

The timex command will display the elapsed, user, and system execution times for a command. Command-line options allow total system activity reports with data items similar to the sar command. To display the system activity created by the ps -ef command, enter # timex -s ps -ef >/dev/null

The iostat command reports processor statistics and all system input and output device statistics

You must have the iostat attribute for sys0 set to true before this command may collect data. Use the lsattr and chdev

commands to list and change this attribute.

# lsattr -El sys0 -a iostat ----Check the iostat attribute

# chdev -l sys0 -a iostat=true ----Change the iostat attribute

Auditing and Security

CONTROLLING SUPERUSER ACCESS

all attempts to become root, whether they succeed or fail, are logged to /var/adm/sulog. To prevent direct logins to the root account from the network, run the command

# chuser rlogin='false' root

AIX provides additional protection by defining the groups who may use the su command to switch to a particular account, and by refusing to permit the use of su to switch to an account. These controls are set in the /etc/security/user file, on a per-user basis:

root:

sugroups = system,security

Set-UID and Set-GID

The root user can also make a file or directory set-UID or set-GID. Normally, when a file is executed, it runs with the UID and GID of the user running the program. However, when a file has been made set-UID, that program is run with the effective user ID of the owner of the file.

chmod u+s /tmp/ksh

Access Control Lists (ACLs)

aclget Prints the ACL set on the specified file

acledit Edits the ACL in the editor specified by the EDITOR environment variable

aclput Assigns the ACL to the file specified, reading from standard input of a separate file

$ aclget /home/datafile

$ acledit /home/datafile

Starting and Stopping Auditing:# /usr/sbin/audit start| shutdown;

To query the status of the auditing subsystem, run the command:# /usr/sbin/audit query

ADDITIONAL SECURITY TOOLS

John the Ripper

John the Ripper is a password-cracking tool. It takes as its input a list of encrypted passwords and tries to determine their unencrypted values.

Distributed Computing Environment (DCE)

System Monitoring and Tuning

The vmstat Command

To examine the virtual memory statistics, use the vmstat command. To tell vmstat to take a snapshot of the system at two-second intervals, and repeat five times: # vmstat 2 5

The vmtune Command

To determine whether the vmtune command is installed and available: # lslpp -lI bos.adt.samples

The vmtune command is located in the /usr/samples/kernel directory.

The iostat Command

If you run the iostat command and see the message “Disk history since boot not available,” this is an indication that the disk activity history is not being retained. To change this: # chdev -l sys0 -a iostat=true

To find out what data is on a given disk, run the lspv -l hdisk# command against that disk. This will show all of the logical volumes that reside on that disk.

Network Performance

The no Command

One very important command to know in AIX networking is the networking options, or no, command.

Whenever the TCP send or receive space parameters are changed, remember that the inetd daemon must be refreshed in order for the changes to take effect.

# no -o tcp_sendspace=65535; # no -o tcp_recvspace=65535; # refresh -s inetd

The netstat Command

To see the statistics for all configured interfaces on a system: # netstat -I

MTU The Maximum Transmission Unit size for the interface

Ipkts Number of packets received (input) on the interface

Ierrs Number of receive errors for the interface

Opkts Number of packets sent (output) to the interface

Coll Number of collisions for the interface

Traditional UNIX Tools

The topas Command

The sar Command

The sar command is used to gather statistical data about CPU usage.

Using sar, we can break those numbers out by processor: # sar -u -P ALL 1 5


Labels

Java (159) Lucene-Solr (110) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (36) Eclipse (34) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (15) Defects (14) Text Mining (14) J2EE (13) Network (13) PowerShell (11) Troubleshooting (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) UIMA (9) html (9) Http Client (8) Maven (8) Problem Solving (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) Lesson Learned (5) Programmer Skills (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) System Design (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts