[CUWiN-Dev] batch upgrades

[Bill Comisky]

I've been using a few scripts to do batch upgrades of the testbed nodes in 
the lab at CNT, and thought I'd share them with the list. I actually just 
found localsrc/node-updater/*, but don't follow how it's meant to be used. 
If there's a better way than how I'm doing it, let me know.

The method I use requires that root be able to ssh in to the node.  I'm 
using an authorized key file called 'root' in the AUTHORIZED_KEY 
directory to set this up password-less.  Buyer beware if you have security 
concerns.  I made a slight change to /sbin/upgrade so that it reads the 
.tar.gz upgrade from STDIN if '-' is used as the upgrade file.  So then I 
can "push" an upgrade by:

cat upgrade.tar.gz | ssh root at nodeIP 'export PATH="/sbin:/usr/sbin:$PATH"; upgrade -'

We've been using svk to mirror the CUWiN svn repository and version our 
local changes.  The attached patch I generated using 'svk diff' from the 
src/boot-image/ subdirectory.  The patch is against svn rev2884; the 
version numbers in the patch comments are our svk repository numbers.

The patch includes changes to:

etc-patches/patch-sshd_config
extras/extraslog
extras/sbin/upgrade

and these new files:

extras/sbin/getversions
extras/sbin/rebootin

Of course you have to setup your own ssh keys.  The last two files are 
helper scripts:

getversions:
Returns the current and next-boot CUWiN release version and device.

rebootin:
reboots the node after a given number of seconds.

Also attached are the 3 server side wrappers scripts to cycle through node 
IPs and run these commands.  For example, on the machine hosting the 
upgrade tarball, the full cycle might look like:

export NODES="10.0.146.86 10.0.244.159 10.0.244.144"

# check current and next-boot CUWiN versions
cuw_getversions $NODES

# push upgrade
cuw_pushupgrade upgrade.tar.gz $NODES

# check again, should show upgrade on 2nd partition
cuw_getversions $NODES

# reboot the nodes 60 seconds, this is just a serial loop
# so leave enough time for the script to ssh in to all the nodes
# on the command line
cuw_rebootnodes 60 $NODES

bill

--
Bill Comisky
bcomisky at pobox.com
-------------- next part --------------
=== etc-patches/patch-sshd_config
==================================================================
--- etc-patches/patch-sshd_config   (/cuw/trunk/src/boot-image)   (revision 2502)
+++ etc-patches/patch-sshd_config   (/cuw/branches/cnt/src/boot-image)   (local)
@@ -0,0 +1,11 @@
+--- etc/ssh/sshd_config	2004-05-05 14:28:49.000000000 -0500
++++ etc/ssh/sshd_config	2004-11-23 15:26:22.000000000 -0600
+@@ -33,7 +33,7 @@
+ 
+ # Slow machines or long keys may require more processing time.
+ LoginGraceTime 600
+-#PermitRootLogin no
++PermitRootLogin yes
+ #StrictModes yes
+ 
+ #RSAAuthentication yes
=== extras/extraslog
==================================================================
--- extras/extraslog   (/cuw/trunk/src/boot-image)   (revision 2502)
+++ extras/extraslog   (/cuw/branches/cnt/src/boot-image)   (local)
@@ -59,6 +59,8 @@
 ./sbin/dhclient-script type=file mode=0555 uname=root gname=wheel
 ./sbin/dhcpselect type=file mode=0555 uname=root gname=wheel
 ./sbin/upgrade type=file mode=0555 uname=root gname=wheel
+./sbin/getversions type=file mode=0555 uname=root gname=wheel
+./sbin/rebootin type=file mode=0555 uname=root gname=wheel
 ./usr type=dir mode=0755 uname=root gname=wheel
 ./usr/sbin type=dir mode=0755 uname=root gname=wheel
 ./usr/sbin/nodeinfo.sh type=file mode=0755 uname=root gname=wheel
=== extras/sbin/upgrade
==================================================================
--- extras/sbin/upgrade   (/cuw/trunk/src/boot-image)   (revision 2502)
+++ extras/sbin/upgrade   (/cuw/branches/cnt/src/boot-image)   (local)
@@ -21,6 +21,11 @@
 usage ()
 {
 	gripe "usage: $0 [-Cfh] user at host:path/to/upgrade.tgz"
+	gripe ""
+	gripe "Upgrades CUWiN to image downloaded from ssh server, or"
+	gripe "from STDIN if - is used in place of the image location."
+	gripe "System will boot to new image a subsequent reboot."
+	gripe ""
 	gripe "	-h	show this helpful message"
 	gripe "	-f	force upgrade, even if node appears to be a CD"
 	gripe "	-C	do not copy existing config file"
@@ -73,11 +78,6 @@
 
 set -u
 
-extract_scp_format="\([^@]*\)@\([^:]*\):\(.*\)"
-user=$(echo $upgrade_path | sed -n -e "s/$extract_scp_format/\1/p")
-host=$(echo $upgrade_path | sed -n -e "s/$extract_scp_format/\2/p")
-tar=$(echo $upgrade_path | sed -n -e "s/$extract_scp_format/\3/p")
-
 current=$(mount | sed -n -e "s/^[^ ]*\([ae]\) on \/ .*/\1/p")
 if [ $current = 'a' ] ; then
 	setactive=1
@@ -92,12 +92,24 @@
 
 echo "Preparing for upgrade on $dev."
 
+if mount | grep -q "on /mnt " ; then
+	umount /mnt
+fi
+
 attempt newfs $rdev
 attempt mount -o async,noatime $dev /mnt
 attempt cd /mnt
 
 echo "Installing the upgrade."
-attempt "ssh $user@$host cat $tar | pax -O -pe -r -z"
+if [ $upgrade_path = '-' ] ; then
+	pax -O -pe -r -z <&0
+else
+	extract_scp_format="\([^@]*\)@\([^:]*\):\(.*\)"
+	user=$(echo $upgrade_path | sed -n -e "s/$extract_scp_format/\1/p")
+	host=$(echo $upgrade_path | sed -n -e "s/$extract_scp_format/\2/p")
+	tar=$(echo $upgrade_path | sed -n -e "s/$extract_scp_format/\3/p")
+	attempt "ssh $user@$host cat $tar | pax -O -pe -r -z"
+fi
 
 echo "Updating etc/fstab"
 fstab=$(mktemp /var/tmp/$(basename $0).fstab.XXXXXX)
=== extras/sbin/getversions
==================================================================
--- extras/sbin/getversions   (/cuw/trunk/src/boot-image)   (revision 2502)
+++ extras/sbin/getversions   (/cuw/branches/cnt/src/boot-image)   (local)
@@ -0,0 +1,54 @@
+#!/bin/sh
+# 
+# Print out partition and CUWiN version, for the current system
+# and for the system at next boot.
+#
+
+[ "$(whoami)" = "root" ] || { 
+	echo "This script is intended to be run as root on a CUW node." 1>&2; 
+	exit 1; 
+}
+
+release_current=`cat /etc/cuwin-release`
+cddev=$(mount | head -n 1 | awk '{print $1}' | grep -q "^/dev/cd")
+if [ $? -eq 0 ] ; then
+   dev_current=$cddev
+   dev_next=$cddev
+   release_next=$release_current
+else
+    
+  # get currently mounted partition
+  dev_current=$(mount | sed -n -e "s/^[^ ]*\([ae]\) on \/ .*/\1/p")
+  if [ "$dev_current" = 'a' ] ; then
+  	  dev_current="/dev/wd0a"
+  else
+  	  dev_current="/dev/wd0e"
+  fi
+  
+  # get partition to be mounted on next boot
+  match='^\([0-9]\): NetBSD';
+  dev_next=$(fdisk | sed -ne "
+  /$match/{
+	   	  N
+		  s/$match.*Active/\1/p
+		  }")
+  if [ "$dev_next" = "0" ] ; then 
+	dev_next="/dev/wd0a"
+  else
+	dev_next="/dev/wd0e"
+  fi
+
+  # get next release version if booting to the other
+  # partition
+  if [ "$dev_current" = "$dev_next" ]; then
+     release_next=$release_current
+  else
+    mount -o ro,async,noatime $dev_next /mnt
+    release_next=`cat /mnt/etc/cuwin-release`
+    umount /mnt
+  fi
+
+fi
+
+echo "  current: on $dev_current running $release_current"
+echo "next boot: on $dev_next running $release_next"

Property changes on: extras/sbin/getversions
___________________________________________________________________
Name: svn:executable
 +*

=== extras/sbin/rebootin
==================================================================
--- extras/sbin/rebootin   (/cuw/trunk/src/boot-image)   (revision 2502)
+++ extras/sbin/rebootin   (/cuw/branches/cnt/src/boot-image)   (local)
@@ -0,0 +1,14 @@
+#!/bin/sh
+
+# redirect all output to /dev/null
+exec >/dev/null 2>&1
+
+if [ $# -lt 1 ] ; then
+   # safe default?
+   sec=600
+else
+   sec=$1
+fi
+
+/bin/sleep $sec
+/sbin/reboot

Property changes on: extras/sbin/rebootin
___________________________________________________________________
Name: svn:executable
 +*

-------------- next part --------------
#!/bin/sh
#

usage() {
    cat 1>&2 <<EOF

Usage: $0 nodeIP [nodeIP]...

Will report /etc/cuwin-release and partition for node
currently and at next boot. Useful for checking software
versions before after 'pushupgrade' and before 'rebootnodes'.

EOF
    exit 1
}

if [ "$#" -lt 1 ]; then
    usage
fi

while [ $# -ne 0 ]; do
  echo "$1"
  ssh root@$1 'export PATH="/sbin:/usr/sbin:$PATH"; getversions'
  echo
  shift
done
-------------- next part --------------
#!/bin/sh
#

usage() {
    cat 1>&2 <<EOF

Usage: $0 cuwin-upgrade.tar.gz nodeIP [nodeIP]...

Pushes a gzip'd CUWiN upgrade tarball to the nodes
given on the command line, running '/sbin/upgrade -'
over ssh on each node as the root user.

EOF
    exit 1
}

if [ "$#" -lt 2 ]; then
    usage
fi

TARBALL=$1
if [ ! -f $TARBALL ]; then
   echo "$TARBALL not found."
   usage
fi
shift

while [ $# -ne 0 ]; do
  echo 
  echo "Pushing $TARBALL to $1:"
  cat $TARBALL | ssh root@$1 'export PATH="/sbin:/usr/sbin:$PATH"; upgrade -'
  shift
done
-------------- next part --------------
#!/bin/sh
#

usage() {
    cat 1>&2 <<EOF

Usage: $0 seconds nodeIP [nodeIP]...

Reboots nodes given on the command line 
after a delay given in seconds.

EOF
    exit 1
}

if [ "$#" -lt 2 ]; then
    usage
fi

REBOOTSECS=$1
shift

while [ $# -ne 0 ]; do
  ssh -n root@$1 "/sbin/rebootin $REBOOTSECS &"
  if [ $? -eq 0 ]; then
     echo "$1 will reboot in $REBOOTSECS seconds"
  else
     echo "Error rebooting $1"
  fi
  shift
done