finding N number of most CPU consuming process in Solaris.

 ps -e -o pcpu -o pid -o user -o args | sort -k 1 | tail -21r

Custom Logrotate in Solaris 10

Here I explain how to configure logadm to rotage any system wide files according to given criteria.
1. Add the corresponding entries in /etc/logadm.conf in below format.
root@server1 # tail -3 /etc/logadm.conf
/var/adm/wtmpx -A 1m -o adm -g adm -m 664 -p 1d -t '$file.old.%Y%m%d_%H%M' -z 1
/var/adm/wtmpx -A 1m -g adm -m 664 -o adm -p 1w -t '$file.old.%Y%m%d_%H%M' -z 5
/var/adm/utmpx -A 1m -g adm -m 664 -o adm -p 1w -t '$file.old.%Y%m%d_%H%M' -z 5
/var/adm/loginlog -A 1m -g sys -m 700 -o root -p 1w -t '$file.old.%Y%m%d_%H%M' -z 5
Explanation for each switch:
-A ->Delete any versions that have not been modified for the amount of time specified by age. Specify age as a number followed by an h (hours), d (days), w(weeks), m (months), or y (years).
-o -> the owner of the newly creating empty file
-g-> the group of newly creating file
-m ->mode of the new empty file (chmod xxx)
-p -> Rotate a log file after the specified time period (period as d, w,  m, y)
-t -> Specify the template to use when renaming log files (Here, wtmpx.old.20101225_0757) (see man logadm for more info)
-z  ->How many copy of rotaged files needs to retain on the system.
-P ->Used by logadm to record the last time the log was rotated in /etc/logadm.conf (no need to set this manually)
2. Once above entries are done, execute logadm -v command to run a logrotation now. Now logadm reads the /etc/logadm.conf file, and for every entry found in that file checks the corresponding log file to see if it should be rotated.
root@server1 # logadm  -v
# loading /etc/logadm.conf
# processing logname: /var/log/syslog
#     using default rotate rules: -s1b -p1w
#     using default template: $file.$n
# processing logname: /var/adm/messages
#     using default rotate rules: -s1b -p1w
#     using default template: $file.$n
# processing logname: /var/cron/log
#     using default expire rule: -C10
# processing logname: /var/lp/logs/lpsched
#     using default rotate rules: -s1b -p1w
# processing logname: /var/fm/fmd/errlog
#     using default expire rule: -C10
#     using default template: $file.$n
# processing logname: /var/fm/fmd/fltlog
#     using default template: $file.$n
# processing logname: smf_logs
#     using default template: $file.$n
# processing logname: /var/adm/pacct
#     using default template: $file.$n
# processing logname: /var/log/pool/poold
#     using default expire rule: -C10
#     using default template: $file.$n
# processing logname: /var/svc/log/system-webconsole:console.log
#     using default rotate rules: -s1b -p1w
#     using default expire rule: -C10
#     using default template: $file.$n
# processing logname: /var/opt/SUNWsasm/log/sasm.log
#     using default template: $file.$n
# processing logname: /var/adm/wtmpx
mkdir -p /var/adm # verify directory exists
mv -f /var/adm/wtmpx /var/adm/wtmpx.old.20101225_1250 # rotate log file
touch /var/adm/wtmpx
chown adm:adm /var/adm/wtmpx
chmod 664 /var/adm/wtmpx
#     recording rotation date Sat Dec 25 12:50:51 2010 for /var/adm/wtmpx
# processing logname: /var/adm/utmpx
mkdir -p /var/adm # verify directory exists
mv -f /var/adm/utmpx /var/adm/utmpx.old.20101225_1250 # rotate log file
touch /var/adm/utmpx
chown adm:adm /var/adm/utmpx
chmod 664 /var/adm/utmpx
#     recording rotation date Sat Dec 25 12:50:51 2010 for /var/adm/utmpx
# processing logname: /var/adm/loginlog
mkdir -p /var/adm # verify directory exists
mv -f /var/adm/loginlog /var/adm/loginlog.old.20101225_1250 # rotate log file
touch /var/adm/loginlog
chown root:sys /var/adm/loginlog
chmod 700 /var/adm/loginlog
#     recording rotation date Sat Dec 25 12:50:51 2010 for /var/adm/loginlog
# writing changes to /etc/logadm.conf
As you can see the last line of above command, once the logadm command successfully run, it do some changes to with -P switch in /etc/logadm.conf file regarding the last update of logrotation.
root@server1 # tail -3 /etc/logadm.conf
/var/adm/wtmpx -A 1m -P 'Sat Dec 25 12:50:51 2010' -g adm -m 664 -o adm -p 1w -t '$file.old.%Y%m%d_%H%M' -z 5
/var/adm/utmpx -A 1m -P 'Sat Dec 25 12:50:51 2010' -g adm -m 664 -o adm -p 1w -t '$file.old.%Y%m%d_%H%M' -z 5
/var/adm/loginlog -A 1m -P 'Sat Dec 25 12:50:51 2010' -g sys -m 700 -o root -p 1w -t '$file.old.%Y%m%d_%H%M' -z 5
List of new files created in /var/adm
root@server1 # ls -ltr /var/adm/*.old*
-rwx------   1 root     sys            0 Dec 25 11:00 /var/adm/loginlog.old.20101225_1250
-rw-r--r--   1 root     bin         3720 Dec 25 15:49 /var/adm/utmpx.old.20101225_1250
-rw-rw-r--   1 adm      adm      8595060 Dec 25 15:51 /var/adm/wtmpx.old.20101225_1250

Growing Sun Cluster File System with new Disks.

Setup Details:
Number of Nodes: 2
Node Name: Node1 and Node2
Cluster: Sun Cluster 3.2
OS: Solaris 9/10

I want to add 300G (100x3) SAN LUNs with one of the cluster mount point (/apps/data).

root@Node2 # df -h|grep d300
/dev/md/apps-ms/dsk/d300   295G   258G    35G    89%    /apps/data

1. Add disks to both systems (shared) in SAN

2. Configure all the fiber channels on both nodes with below steps.

root@Node1 #  cfgadm -al|grep fc
c4                             fc-fabric    connected    configured   unknown
c5                             fc           connected    unconfigured unknown
c6                             fc-fabric    connected    configured   unknown
c7                             fc           connected    unconfigured unknown
root@Node1 # cfgadm -c configure c4 c5 c6 c7

3. Run devfsadmin to configure new devices

root@Node1 #devfsadm –C

(Repeat steps 2 and 3 in all cluster nodes)
4. Run format command to list all the disks, the newly configred disk can be seen at top of format as below (if the disk not labeled already)

root@Node1 # format
Searching for disks...done

c8t6005076305FFC08C0000000000000103d0: configured with capacity of 99.98GB
c8t6005076305FFC08C0000000000000104d0: configured with capacity of 99.98GB
c8t6005076305FFC08C0000000000000120d0: configured with capacity of 99.98GB

5. Format each disk to create a partition as below.

s7 ->100mb (this 100 mb is reseverd for metadb creation. Not mandatory)
s0 -> remaining space.

6. Create corresponding cluster devices (global device path) using scgdevs command.

root@Node2 # scgdevs
Configuring DID devices
did instance 95 created.
did subpath Node2:/dev/rdsk/c8t6005076305FFC08C0000000000000120d0 created for instance 95.
did instance 96 created.
did subpath Node2:/dev/rdsk/c8t6005076305FFC08C0000000000000104d0 created for instance 96.
did instance 97 created.
did subpath Node2:/dev/rdsk/c8t6005076305FFC08C0000000000000103d0 created for instance 97.
Configuring the /dev/global directory (global devices)
obtaining access to all attached disks
(above command resulted in createing d95, d96, d97 devices)

7. Confirm this devices are available on both nodes. There must be same devices with each hostname as given below.

root@Node2 # scdidadm -L|egrep 'd95|d96|d97'
95       Node2:/dev/rdsk/c8t6005076305FFC08C0000000000000120d0 /dev/did/rdsk/d95
95       Node1:/dev/rdsk/c8t6005076305FFC08C0000000000000120d0 /dev/did/rdsk/d95
96       Node2:/dev/rdsk/c8t6005076305FFC08C0000000000000104d0 /dev/did/rdsk/d96
96       Node1:/dev/rdsk/c8t6005076305FFC08C0000000000000104d0 /dev/did/rdsk/d96
97       Node2:/dev/rdsk/c8t6005076305FFC08C0000000000000103d0 /dev/did/rdsk/d97
97       Node1:/dev/rdsk/c8t6005076305FFC08C0000000000000103d0 /dev/did/rdsk/d97

Following steps must be done on the system which has the ownership of this metaset (metaset -s apps-ms and confirm who is the owner)

8. Adding all the three devices with curresponding metaset (apps-ms)

root@Node2 # metaset -s apps-ms -a /dev/did/rdsk/d95 /dev/did/rdsk/d96 /dev/did/rdsk/d97

9. Attach this devices with specific meta devices (here it’s d300) using metattach command.

root@Node2 # metattach -s apps-ms d300 /dev/did/rdsk/d95s0 /dev/did/rdsk/d96s0 /dev/did/rdsk/d97s0
apps-ms/d300: components are attached

10. Confirm the devices are attached properly using below command.

root@Node2 # metastat -s apps-ms -p d300 apps-ms/d300 2 3 d6s0 d7s0 d8s0 -i 32b \
         3 d95s0 d96s0 d97s0 -i 32b

11. Once the above result is confirmed, file system can be grown using below command.

root@Node2 # growfs -M /apps/data /dev/md/apps-ms/rdsk/d300
/dev/md/apps-ms/rdsk/d300:    1257996288 sectors in 76782 cylinders of 64 tracks, 256 sectors
        614256.0MB in 12797 cyl groups (6 c/g, 48.00MB/g, 5824 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 98592, 197152, 295712, 394272, 492832, 591392, 689952, 788512, 887072,
Initializing cylinder groups:
...............................................................................
...............................................................................
...............................................................................
..................
super-block backups for last 10 cylinder groups at:
 1257026336, 1257124896, 1257223456, 1257322016, 1257420576, 1257519136,
 1257617696, 1257716256, 1257814816, 1257913376,

12. After successfull execution of above command, the file system has been grow. Now its around 600G.

 root@Node2 # df -h|grep d300
/dev/md/apps-ms/dsk/d300   591G   258G   330G    44%    /apps/data

13. Below is the corresponding logs generated in /var/adm/messages during above activity.

System Logs:
Dec 21 10:07:21 Node1 Cluster.devices.did: [ID 287043 daemon.notice] did subpath /dev/rdsk/c8t6005076305FFC08C0000000000000120d0s2 created for instance 95.
Dec 21 10:07:22 Node1 Cluster.devices.did: [ID 536626 daemon.notice] did subpath /dev/rdsk/c8t6005076305FFC08C0000000000000104d0s2 created for instance 96.
Dec 21 10:07:22 Node1 Cluster.devices.did: [ID 624417 daemon.notice] did subpath /dev/rdsk/c8t6005076305FFC08C0000000000000103d0s2 created for instance 97.
Dec 21 10:07:22 Node1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d95s0 is set to MONITORED
Dec 21 10:07:22 Node1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d96s0 is set to MONITORED
Dec 21 10:07:22 Node1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d96s0 has changed to OK
Dec 21 10:07:22 Node1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d95s0 has changed to OK
Dec 21 10:07:22 Node1 Cluster.scdpmd: [ID 922726 daemon.notice] The status of device: /dev/did/rdsk/d97s0 is set to MONITORED
Dec 21 10:07:22 Node1 Cluster.scdpmd: [ID 489913 daemon.notice] The state of the path to device: /dev/did/rdsk/d97s0 has changed to OK
Dec 21 10:07:39 Node1 Cluster.devices.did: [ID 466922 daemon.notice] obtaining access to all attached disks

Network troubleshooting from OBP (OK prompt)

For listing all the available network interfaces on the system, use below command from ok prompt.
{0} ok watch-net
Timed out waiting for Autonegotiation to complete
Check cable and try again
Link Down
{0} ok watch-net-all
/pci@3,700000/network@0,1 ----------------------> path to interface
Timed out waiting for Autonegotation to complete
Check cable and try again ----------------------> No cable is connected/active
Link Down
Timed out waiting for Autonegotation to complete
Check cable and try again
Link Down
Timed out waiting for Autonegotation to complete
Check cable and try again
Link Down
Timed out waiting for Autonegotation to complete
Check cable and try again
Link Down
/pci@3,700000/network@0   -------------------> path to interface

1000 Mbps full duplex  Link up
Looking for Ethernet Packets.
'.' is a Good Packet.  'X' is a Bad Packet.
Type any key to stop.
40 42 42 40 42 42 40 40 42 42 40 1ed 42 42 40 42 40 42 40 42 42 40 42 40 42 40 42 42 40 42 40 42 40 42 42 40 40 42 42 40 40 42 42 40 42 42 40 40 42 42 40 42 42 40 42 40 42 40 42 42 40 42 40 42 40 42 42 40
/pci@2,600000/network@0,1
1000 Mbps full duplex  Link up
Looking for Ethernet Packets.
'.' is a Good Packet.  'X' is a Bad Packet.
Type any key to stop.
40 44 44 44 40 7a 40 f7 40 40 40 1ed 40 40
/pci@2,600000/network@0
1000 Mbps full duplex  Link up
Looking for Ethernet Packets.
'.' is a Good Packet.  'X' is a Bad Packet.
Type any key to stop.
42 40 42 42 40 42 40 42
/pci@0,600000/pci@0/pci@8/pci@0/network@2,1
Timed out waiting for Autonegotiation to complete
Check cable and try again
Link Down
/pci@0,600000/pci@0/pci@8/pci@0/network@2
Timed out waiting for Autonegotiation to complete
Check cable and try again
Link Down
For getting the details about each interfaces, go to the corresponding path using cd comammnd, and run the properties as below
{0} ok cd /pci@3,700000/network@0
{0} ok .properties
mac-address              00 15 17 3b xx xx
link-clock               auto
duplex                   auto
speed                    auto
status                   okay
assigned-addresses       82020010 00000000 00100000 00000000 00020000
                         82020014 00000000 00120000 00000000 00020000
                         81020018 00000000 00000300 00000000 00000020
                         82020030 00000000 00140000 00000000 00020000
phy-type                 mif
board-model              501-7289
version                  Sun PCI-E 1G Ethernet UTP Adapter FCode 1.10 06/11/02
model                    SUNW,pcie-northstar
d-fru-len                00000800
d-fru-off                00006800
d-fru-dev                eeprom
s-fru-len                00000800
s-fru-off                00006000
s-fru-dev                eeprom
compatible               pciex8086,105e.108e.125e.6
                         pciex8086,105e.108e.125e
                         pciex108e,125e
                         pciex8086,105e.6
                         pciex8086,105e
                         pciexclass,020000
                         pciexclass,0200
reg                      00020000 00000000 00000000 00000000 00000000
                         02020010 00000000 00000000 00000000 00020000
                         02020030 00000000 00000000 00000000 00020000
max-frame-size           00010000
address-bits             00000030
device_type              network
name                     network
local-mac-address        00 15 17 3b xx xx --> MAC Address for /pci@3,700000/network@0
fcode-rom-offset         0000e000
interrupts               00000001
cache-line-size          00000010
class-code               00020000
subsystem-id             0000125e
subsystem-vendor-id      0000108e
revision-id              00000006
device-id                0000105e
vendor-id                00008086
{0} ok cd /pci@2,600000/network@0,1
{0} ok .properties
status                   okay
assigned-addresses       82020110 00000000 00160000 00000000 00020000
                         82020114 00000000 00180000 00000000 00020000
                         81020118 00000000 00000320 00000000 00000020
                         82020130 00000000 001a0000 00000000 00020000
phy-type                 mif
board-model              501-7289
version                  Sun PCI-E 1G Ethernet UTP Adapter FCode 1.10 06/11/02
model                    SUNW,pcie-northstar
d-fru-len                00000800
d-fru-off                00006800
d-fru-dev                eeprom
s-fru-len                00000800
s-fru-off                00006000
s-fru-dev                eeprom
compatible               pciex8086,105e.108e.125e.6
                         pciex8086,105e.108e.125e
                         pciex108e,125e
                         pciex8086,105e.6
                         pciex8086,105e
                         pciexclass,020000
                         pciexclass,0200
reg                      00020100 00000000 00000000 00000000 00000000
                         02020110 00000000 00000000 00000000 00020000
                         02020130 00000000 00000000 00000000 00020000
max-frame-size           00010000
address-bits             00000030
device_type              network
name                     network
local-mac-address        00 15 17 3d xx xx --> MAC Address for /pci@2,600000/network@0,1
fcode-rom-offset         0000e000
interrupts               00000002
cache-line-size          00000010
class-code               00020000
subsystem-id             0000125e
subsystem-vendor-id      0000108e
revision-id              00000006
device-id                0000105e
vendor-id                00008086
{0} ok cd /pci@2,600000/network@0
{0} ok .properties
status                   okay
assigned-addresses       82020010 00000000 00100000 00000000 00020000
                         82020014 00000000 00120000 00000000 00020000
                         81020018 00000000 00000300 00000000 00000020
                         82020030 00000000 00140000 00000000 00020000
phy-type                 mif
board-model              501-7289
version                  Sun PCI-E 1G Ethernet UTP Adapter FCode 1.10 06/11/02
model                    SUNW,pcie-northstar
d-fru-len                00000800
d-fru-off                00006800
d-fru-dev                eeprom
s-fru-len                00000800
s-fru-off                00006000
s-fru-dev                eeprom
compatible               pciex8086,105e.108e.125e.6
                         pciex8086,105e.108e.125e
                         pciex108e,125e
                         pciex8086,105e.6
                         pciex8086,105e
                         pciexclass,020000
                         pciexclass,0200
reg                      00020000 00000000 00000000 00000000 00000000
                         02020010 00000000 00000000 00000000 00020000
                         02020030 00000000 00000000 00000000 00020000
max-frame-size           00010000
address-bits             00000030
device_type              network
name                     network
local-mac-address        00 15 17 3d xx xx --> MAC Address for  /pci@2,600000/network@0
fcode-rom-offset         0000e000
interrupts               00000001
cache-line-size          00000010
class-code               00020000
subsystem-id             0000125e
subsystem-vendor-id      0000108e
revision-id              00000006
device-id                0000105e
vendor-id                0000808