IPMI Configuration on Rocks Cluster

The following is a step by step guide on setting up IPMI management on a Rocks Cluster. It is based on several great posts on Rocks Mailing list and Rocks userguide presentation

Update: On Rocks Mailing list Philip Papadopoulos mentioned a Python Script that will automatically add IPMI interface by matching the last 2 octets to that of node: Script

1. Create the IPMI Network

rocks add network ipmi subnet=10.2.0.0 netmask=255.255.0.0 servedns=true

2. Add access on head node

The IPMI network is sharing the same vlan as data, so I will multihome the Frontend with a second IP on IPMI subnet

This adds it to Rocks config, but it appears to have no effect so I also ran it manually

rocks add host interface localhost iface=eth0:1 ip=10.2.1.1 subnet=ipmi name=ipmi-frontend

Manually:

ifcfg eth0:1 add 10.2.1.1/16

Add a script to make it start at reboot

Just copy /etc/sysconfig/network-scripts/ifcfg-eth0 to ifcfg-eth0:1 and change the device and IP address.

DEVICE=eth0:1
HWADDR=00:xx:xx:xx:xx:xx
IPADDR=10.2.1.1
NETMASK=255.255.0.0
BOOTPROTO=static
ONBOOT=yes
MTU=1500

3. Set Password for IPMI

You can set a custom password for the IPMI system or use the default of admin

rocks set attr ipmi_password "password" shadow=yes

4. Add IPMI info for each node

Updated way:
Download the latest addInterface and ScarScript.py and run the following to add matching IP for each node.
If you are running Rocks 6.0/6.1 addInterface will not work for nodes whose card is not card eth0 (e.g. instead is em1). To fix this change line 67 to

"rocks list host interface %s | grep 'private ' | awk ' {print $4}'" %
for n in {0..6}; do
./addInterface compute-0-$n ipmi ipmi ipmi-0-$n
rocks set host interface channel compute-0-$n iface=ipmi channel=1;
done

Old Way:

node=0
for n in {254..245}; do
rocks add host interface compute-0-$node iface=ipmi ip=10.2.255.$n \
module=1 subnet=ipmi name=ipmi-0-$node;
rocks set host interface channel compute-0-$node iface=ipmi channel=1;
let node=$node+1;
done

Sync it

rocks sync host network

If you have every had DRAC set to use DHCP it needs to be set to static.

rocks run host "ipmitool lan set 1 ipsrc static" collate=yes

I also wanted to set the default gateway which is not set by the generated script and it will complain about network config if not set.

rocks run host "ipmitool lan set 1 defgw ipaddr 10.2.1.1" collate=yes

5. IPMI root Password

When I tried to connect using ipmitool and set password it reported NULL account disabled. To get around this is used the root account (id=2) and set a password for this.

rocks run host "ipmitool user set password 2 <password>"

6. Run the generated script on each node

rocks run host "sh /etc/sysconfig/ipmi"

7. Testing

ipmitool -I lan -H ipmi-0-0.ipmi -U root -P <password> chassis status

About James Rudd

Network Administrator at Sydney Boys High School
This entry was posted in Linux and tagged , , , , , . Bookmark the permalink.

Leave a Reply