VCS - Link Based IPMP Configuration

Interface configuration files.


root@node055 # cat /etc/hostname.nxge3
node055 netmask + broadcast + group production up

root@node055 # cat /etc/hostname.nxge9
group production standby up



VCS Configuration File Entry (main.cf)



group Multinic (
        SystemList = { node055 = 0, node056 = 1, node057 = 2, node058 = 3,
                 node059 = 4 }
        Parallel = 1
        AutoStartList = { node055, node056, node057, node058, node059 }
        )

        MultiNICB MNICB-Res (
                Critical = 0
                Device = { nxge3 = "", nxge9 = "" }
                UseMpathd = 1
                GroupName = production
                ConfigCheck = 0
                )

        Phantom Phantom (
                Critical = 0
                )

Linux IP Multipath configuration (bonding)

Check all your network interfaces using ethtool command

[root@MyLinux ~]# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes
[root@MyLinux ~]# ethtool eth1
Settings for eth1:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supports auto-negotiation: Yes
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised auto-negotiation: Yes
        Speed: 1000Mb/s
        Duplex: Full
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: g
        Wake-on: g
        Link detected: yes

Check all the requiured kernel modules are available(bonding and mii)
[root@MyLinux ~]# modprobe -l | grep bonding
/lib/modules/2.6.18-308.el5/kernel/drivers/net/bonding/bonding.ko

[root@MyLinux ~]# modprobe -l | grep mii
/lib/modules/2.6.18-308.el5/kernel/drivers/net/mii.ko

Editing the modprobe.conf file. Append below line in modprobe.conf
[root@MyLinux etc]# cat modprobe.conf
alias bond0 bonding
options bond0 mode=1 miimon=100

Where

mode=0 (balance-rr)
Round-robin policy: Transmit packets in sequential order from the first available slave through the last. This mode provides load balancing and fault tolerance.

mode=1 (active-backup)
Active-backup policy: Only one slave in the bond is active. A different slave becomes active if, and only if, the active slave fails. The bond's MAC address is externally visible on only one port (network adapter) to avoid confusing the switch. This mode provides fault tolerance. The primary option affects the behavior of this mode.

mode=2 (balance-xor)
XOR policy: Transmit based on [(source MAC address XOR'd with destination MAC address) modulo slave count]. This selects the same slave for each destination MAC address. This mode provides load balancing and fault tolerance.

mode=3 (broadcast)
Broadcast policy: transmits everything on all slave interfaces. This mode provides fault tolerance.

mode=4 (802.3ad)
IEEE 802.3ad Dynamic link aggregation. Creates aggregation groups that share the same speed and duplex settings. Utilizes all slaves in the active aggregator according to the 802.3ad specification.


Load both kernel modules and confirm the file edit by below commands

[root@MyLinux etc]# modprobe bonding
[root@MyLinux etc]# modprobe mii


Create a new network configuration file for bond0 and edit eth0 and eth1 accordingly under /etc/sysconfig/network-scripts/ directory

ifcfg-bond0

[root@MyLinux network-scripts]# cat ifcfg-bond0
DEVICE=bond0
BOOTPROTO=none
ONBOOT=yes
NETMASK=255.255.254.0
IPADDR=192.168.10.139
USERCTL=no
GATEWAY=192.168.10.1
TYPE=Ethernet
IPV6INIT=no
PEERDNS=yes
[root@MyLinux network-scripts]#

ifcfg-eth0

[root@MyLinux network-scripts]# cat ifcfg-eth0
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth0
BOOTPROTO=none
#HWADDR=e4:1f:13:1e:c4:00
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
TYPE=Ethernet

ifcfg-eth1

[root@MyLinux network-scripts]# cat ifcfg-eth1
# Broadcom Corporation NetXtreme II BCM5709 Gigabit Ethernet
DEVICE=eth1
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
USERCTL=no
TYPE=Ethernet

Restart network services now.
[root@MyLinux etc]# cd /etc/sysconfig/network-scripts/
[root@MyLinux network-scripts]# /etc/init.d/network restart
Shutting down interface eth0:                              [  OK  ]
Shutting down interface eth1:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Bringing up loopback interface:                            [  OK  ]
Bringing up interface bond0:                               [  OK  ]

Check the bond status using below commands

[root@MyLinux ~]#  cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.4.0-1 (October 7, 2008)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e4:1f:13:1e:c4:00

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: e4:1f:13:1e:c4:02


[root@MyLinux ~]# ifconfig -a
bond0     Link encap:Ethernet  HWaddr E4:1F:13:1E:C4:00
          inet addr:192.168.10.139  Bcast:10.64.209.255  Mask:255.255.254.0
          inet6 addr: fe80::e61f:13ff:fe1e:c400/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:2420 errors:0 dropped:0 overruns:0 frame:0
          TX packets:530 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:210110 (205.1 KiB)  TX bytes:351640 (343.3 KiB)

eth0      Link encap:Ethernet  HWaddr E4:1F:13:1E:C4:00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:545 errors:0 dropped:0 overruns:0 frame:0
          TX packets:530 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:53341 (52.0 KiB)  TX bytes:351640 (343.3 KiB)
          Interrupt:98 Memory:f2000000-f2012800

eth1      Link encap:Ethernet  HWaddr E4:1F:13:1E:C4:00
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1875 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:156769 (153.0 KiB)  TX bytes:0 (0.0 b)
          Interrupt:193 Memory:f4000000-f4012800