CAU Local VM Script

Cluster Aware Updating (CAU) introduced in Server 2012 allows you to update your Fail-Over cluster by draining all the Highly Available (HA) roles from the node, performing the updates and then restoring the roles.

While this works great for Clustered HA VMs on a Hyper-V cluster, it appears CUA completely ignores any unclustered VMs on the Nodes. We have non-HA VMs on our Hyper-V Hosts that act as nodes in a File server Fail-Over Cluster and Exchange DAG groups. When CUA runs it ignores the shutdown settings of these VMs and just powers them off, this can causes loss of data in nodes and causes the cluster to have to bring up a down resource rather than just transfer the role.

To fix this we had to create scripts that shut down all the local VMs and then start them back up. These can then be set in the PreUpdateScript and PostUpdateScript profile settings for CUA. I stored the scripts on the CSV volume in C:\ClusterStorage\Volume1\Hyper-V\CAU\. These could then be added as a local file in CAU configuration wizard.

PreUpdateScript = ShutdownLocal.ps1

# VM Delay
$CheckDelay = 30

# Local VMs
$LocalVMs = Get-VM | Where-Object { ($_.IsClustered -eq $false) -and ($_.State -ne "Off")} 

$Node = $env:computername
$LogFile = "C:\ClusterStorage\Volume1\Hyper-V\CAU\Log\$(get-date -f yyyy-MM-dd)-$Node.log"

(Get-Date -Format s) + " CAU Shutdown Script" >> $LogFile

#Shutdown Each VM
ForEach ($VM in $LocalVMs) {
    $VMName = $VM.Name
    # Check State
    If ($VM.State -ne "Off") {
        If ($VM.AutomaticStopAction -eq "ShutDown") {
            # Change State
            (Get-Date -Format s) + " Shutting Down $VMName" >> $LogFile
            Stop-VM -AsJob -VM $VM  #-WhatIf
        } 
        ElseIf ($VM.AutomaticStopAction -eq "Save") {
            # Change State
            (Get-Date -Format s) + " Saving $VMName" >> $LogFile
            Stop-VM -AsJob -VM $VM -Save  #-WhatIf
        } 
    }
}

# Wait for all VMs to Shutdown
$i = 0
Do {
    $VMon = 0
    $i++
    $TimeWait = $CheckDelay * $i
    Start-Sleep -Seconds $CheckDelay
    (Get-Date -Format s) + " Waited $TimeWait seconds for:" >> $LogFile
    ForEach ($VM in $LocalVMs) {      
        $VMName = $VM.Name
        # Check State
        If ((Get-VM -Name $VMName).State -ne "Off" -and (Get-VM -Name $VMName).State -ne "Saved") {
            # Report State
            (Get-Date -Format s) + " -$VMName" >> $LogFile
            $VMon++
        }
    }
}
Until ( ($i -gt 10) -or ($VMon -eq 0) )

(Get-Date -Format s) + " CAU Shutdown Script Completed" >> $LogFile

PostUpdateScript = StartLocal.ps1


$Node = $env:computername
$LogFile = "C:\ClusterStorage\Volume1\Hyper-V\CAU\Log\$(get-date -f yyyy-MM-dd)-$Node.log"

$PauseDelay = 300
# Local VMs
$LocalVMs = Get-VM | 
  Where-Object { ($_.IsClustered -eq $false) -and  ($_.AutomaticStartAction -like 'Start*') -and ($_.State -ne "Running")} | 
  Sort-Object AutomaticStartDelay

(Get-Date -Format s) + " CAU Startup Script" >> $LogFile

$DelayElapsed = 0
#Start Each VM
ForEach ($VM in $LocalVMs) {
    $VMName = $VM.Name
    $VMDelay = $VM.AutomaticStartDelay
    # Check State
    If ($VM.State -ne "Running") {
        $VMDiff = $VMDelay - $DelayElapsed
        If ($VMDiff -gt 0) { Start-Sleep -Seconds $VMDiff }
        $DelayElapsed += $VMDiff
        # Change State
        (Get-Date -Format s) + " Starting $VMName" >> $LogFile
        Start-VM -AsJob -VM $VM  #-WhatIf
    }
}
# Delay as will immediately start shutting down next node
(Get-Date -Format s) + " VMs Started waiting $PauseDelay seconds for clusters to stabilise" >> $LogFile
Start-Sleep -Seconds $PauseDelay
(Get-Date -Format s) + " CAU Startup Script Completed" >> $LogFile
Posted in Windows | Tagged , , , , | Leave a comment

Updating Moodle table comments

Recently we updated Moodle to 2.6, after doing so I compared the SQL table structure of our upgraded site database against a clean install with the same addons. I found quite a few differences in table structure (smallint to tinyint, etc) for the 3rd party addons, and most core tables were missing their table comments.

To fix this I wrote the following perl script, it takes in a SQL structure dump from PhpMyAdmin and generates a list of alter commands to apply all the table comments in that file. This can then be run against your actual Moodle DB.


#!C:\Perl\bin\perl.exe

# Designed to read in SQL structure export and generate a list of alter commands to set comments from those tables

use strict; use warnings;

my $fileName = $ARGV[0];

open F, "< $fileName" or die "Can't open $fileName : $!"; my %comments = (); my $currentTable; while (my $line = ){
if ($line =~ /CREATE TABLE `(.+)`/) {
# Set current table name
# CREATE TABLE `mdl_assign`
$currentTable = $1;
}
if ($line =~ /COMMENT='(.+)'/) {
# Store table comment
$comments{ $currentTable } = $1;
}
}
close F;

foreach my $table ( sort keys %comments ) {
print "ALTER TABLE `" . $table . "` COMMENT = '" . $comments{$table} . "';\n";
}

Posted in Moodle | Leave a comment

Moodle Assignment Editor showing HTML instead of GUI editor

Recently one of teachers noticed that some Assignments that had been copied between courses would show the assignment introduction as a plaint ext box with html markup visible.

I tracked this down to Moodle recording the wrong format for the intro. To fix all instances of this in the database I ran the following query:

UPDATE `mdl_assignment`
SET `introformat` = 1
WHERE `introformat` = 0 AND `intro` like '%<p%'

This updated the format for the 1400 or so assignments that had been copied and had this problem.

Posted in Moodle | Leave a comment

IPMI Configuration on Rocks Cluster

The following is a step by step guide on setting up IPMI management on a Rocks Cluster. It is based on several great posts on Rocks Mailing list and Rocks userguide presentation

Update: On Rocks Mailing list Philip Papadopoulos mentioned a Python Script that will automatically add IPMI interface by matching the last 2 octets to that of node: Script

1. Create the IPMI Network

rocks add network ipmi subnet=10.2.0.0 netmask=255.255.0.0 servedns=true

2. Add access on head node

The IPMI network is sharing the same vlan as data, so I will multihome the Frontend with a second IP on IPMI subnet

This adds it to Rocks config, but it appears to have no effect so I also ran it manually

rocks add host interface localhost iface=eth0:1 ip=10.2.1.1 subnet=ipmi name=ipmi-frontend

Manually:

ifcfg eth0:1 add 10.2.1.1/16

Add a script to make it start at reboot

Just copy /etc/sysconfig/network-scripts/ifcfg-eth0 to ifcfg-eth0:1 and change the device and IP address.

DEVICE=eth0:1
HWADDR=00:xx:xx:xx:xx:xx
IPADDR=10.2.1.1
NETMASK=255.255.0.0
BOOTPROTO=static
ONBOOT=yes
MTU=1500

3. Set Password for IPMI

You can set a custom password for the IPMI system or use the default of admin

rocks set attr ipmi_password "password" shadow=yes

4. Add IPMI info for each node

Updated way:
Download the latest addInterface and ScarScript.py and run the following to add matching IP for each node.
If you are running Rocks 6.0/6.1 addInterface will not work for nodes whose card is not card eth0 (e.g. instead is em1). To fix this change line 67 to

"rocks list host interface %s | grep 'private ' | awk ' {print $4}'" %
for n in {0..6}; do
./addInterface compute-0-$n ipmi ipmi ipmi-0-$n
rocks set host interface channel compute-0-$n iface=ipmi channel=1;
done

Old Way:

node=0
for n in {254..245}; do
rocks add host interface compute-0-$node iface=ipmi ip=10.2.255.$n \
module=1 subnet=ipmi name=ipmi-0-$node;
rocks set host interface channel compute-0-$node iface=ipmi channel=1;
let node=$node+1;
done

Sync it

rocks sync host network

If you have every had DRAC set to use DHCP it needs to be set to static.

rocks run host "ipmitool lan set 1 ipsrc static" collate=yes

I also wanted to set the default gateway which is not set by the generated script and it will complain about network config if not set.

rocks run host "ipmitool lan set 1 defgw ipaddr 10.2.1.1" collate=yes

5. IPMI root Password

When I tried to connect using ipmitool and set password it reported NULL account disabled. To get around this is used the root account (id=2) and set a password for this.

rocks run host "ipmitool user set password 2 <password>"

6. Run the generated script on each node

rocks run host "sh /etc/sysconfig/ipmi"

7. Testing

ipmitool -I lan -H ipmi-0-0.ipmi -U root -P <password> chassis status

Posted in Linux | Tagged , , , , , | Leave a comment