TL;DR
You can use Docker to build a staticly linked ipmitool that will
run on VMWare ESXi
5.x to
confgure a Dell iDRAC's network settings. Read on for details of how
this works.
Motivation
Imagine that you have a set of Dell servers running VMWare ESXi colocated
in a data centre. And that you are concerned that you might need to
interact with the servers before they have fully booted -- perhaps when
they are not booted -- so that ssh and the VMWare management
interfaces cannot be used. In those instances it would be very useful
to be able to interact with the console of the remote machines, and at
least reassure yourself that they are still booting.
If your Dell Servers were bought with Dell iDRAC support, then you have the hardware you need -- you just need to cable it up to some out of band network (ideally via some independent Internet connection so you do not rely on your own routers having booted), and configure the iDRAC network interface so you can reach it via that out of band network.
The cabling part is fairly easy -- give or take a trip to the data centre. But by default the configuration of the iDRAC can be done on the server only by interrupting the boot process at the appropriate point. Which would mean rebooting the ESXi servers. The production ESXi servers. Yes it is a cluster, but that is a lot of shuffling VMs around to free up one host at a time so it can be rebooted, which is a lot of time in the data centre.
On Linux and Microsoft Windows, Dell provide a racadm
tool
which can be used to configure basic features of the Dell iDRAC --
enough that you can then get in via the web interface and configure
the rest. The only trick is making sure that you use the correct
version for your iDRAC card -- inexplicably Dell appear to have not
only changing the configuration interface between versions but also
provided separate binaries for each version (rather than auto-detecting
the installed card and supporting all the protocols in one program).
With the wrong version for your iDRAC card, you will be very confused
why it does not seem to work.
Unfortuantely there is no VMWare ESXi version of the local racadm (at
least that I have been able to find). And the remote racadm needs
the IP address and username/password of the iDRAC to talk to... which
brings us back to the first problem.
But conveniently the Dell iDRAC is based on IPMI
2.0, which means
that
IPMI
tools can be used to access some of the functionality. (This seems
to have been true since the Dell DRAC
5;
there is even a Dell guide to using ipmitool with the older
PowerEdge
servers.)
ipmitool
On Linux, one of the standard tools for interacting with the IPMI
(locally and remotely) is
ipmitool (which
unfortunately seems to still be hosted on the
increasingly
untrustworthy
SourceForge -- but source tarballs are also available via major Linux
vendors). Ryan Chapman discovered that a "statically" linked Linux
ipmitool would run on VMWare ESXi
4.1, and John found this still
works on VMWare ESXi
5.1,
as did Mitchell
Turner.
The main requirements for using a Linux ipmitool under VMWare
ESXi seem to be (a) ensuring it is built on a Linux release that
is approximately old enough to match the libc and kernel versions
of the VMWare ESXi host, (b) buliding a 32-bit binary, and (c)
making sure it is statically linked. (Possibly either matching
libc versions or statically linked would be sufficient, but aiming
for both maximises the chances of it working.)
On VMWare ESXi 5.0 (still in use due to being a well stabilised
combination of software and drivers, pending a jump to ESXi 6.x),
the libc there was:
~# /lib/libc.so.6
GNU C Library vmware release version 2.5, by Roland McGrath et al.
[...]
Compiled by GNU CC version 4.1.2 20070626 (Red Hat 4.1.2-14).
Compiled on a Linux 2.6.9 system on 2015-01-29.
compiled with a compiler and on a kernel version which are roughly equivalent to CentOS 4, of which the last release was CentOS 4.9, and support ended about three years ago.
32-bit CentOS 4.9 build environment
So we wanted a 32-bit CentOS 4.9 build environment. Since I did not
fancy doing a full install of an ancient Linux system (into a VM)
just to build a single tool I went looking for a more convenient
answer. It turns out that while the official CentOS Docker
images are only 64-bit
(x86_64) and only start with CentOS 5, Brian
Lalor has produced an unofficial Docker
build script to make a base CentOS 4.9
image. Unfortuantely
that too was 64-bit (x86_64) but with some simple tweaks it is
possible to get it to build a 32-bit (i386) CentOS 4.9 base image:
git clone https://github.com/blalor/docker-centos4-base.git
cd docker-centos4-base
vi build.sh # Change arch from x86_64 to i386, comment out EPEL
docker run --privileged -i -t -v $PWD:/srv centos:centos6 /srv/build.sh i386
docker build -t centos4_i386 .
The docker run produces a centos49.tar.gz chroot tarball (a bit
like debootstrap on Debian),
and the docker build builds a Docker image with just that tarball
unpacked inside it. (My docker-centos4-base build.sh
script
takes the architecture as a command line parameter; if I get a
chance to make that a bit less hacky I will try to push it back
upstream.)
CentOS 4.9 build environment fixups
From there, it just a matter of fixing up the 32-bit CentOS environment
to force it to install 32-bit binaries (by default it will install
binaries for
$basearch
which will be 64-bit), and then installing the minimal ipmitool build
dependencies. To do this by hand:
docker run -it centos4_i386 /bin/bash
cat /etc/yum.repos.d/CentOS-Base.repo |
sed 's/\$basearch/i386/; s/\]/-i386]/;' |
tee /etc/yum.repos.d/CentOS-i386.repo
sed -i '/protect=1/a enabled=0' /etc/yum.repos.d/CentOS-Base.repo
(these examples chosen to avoid IO redirection as the angle brackets
conflict with the HTML markup :-( ). And then it is necesssary to
manually install the CentOS 4 yum key before installing packages:
curl -L -o /tmp/RPM-GPG-KEY-centos4 http://mirror.centos.org/centos/RPM-GPG-KEY-centos4
sha256sum /tmp/RPM-GPG-KEY-centos4
# Should be:
# 57821109aeb5f27805b80be03e31b1d978f3b7c7464c684169b1d341ff4d7021 /tmp/RPM-GPG-KEY-centos4
rpm --import /tmp/RPM-GPG-KEY-centos4
Building static ipmitool
NOTE: If you have Docker
running,
you can can use Docker to build a staticly linked ipmitool that
will run on VMWare ESXi
5.x
automatically. The Dockerfile automates the build environment tidying
up above and all the build steps below. The build steps below are
listed for completeness in case you want to do it by hand in some other
environment.
Building ipmitool is fairly simple, thanks to it having very few
dependencies. It just needs gcc4 and make:
yum -y install gcc4 make
First download the source archive, eg:
curl -L -o ipmitool_1.8.14.tar.gz http://http.debian.net/debian/pool/main/i/ipmitool/ipmitool_1.8.14.orig.tar.gz
(this one from Debian's archive, which I trust more than Sourceforge now), and then verify the checksum:
sha256sum ipmitool_1.8.14.tar.gz
# Should be:
# fe9bce4587f83fe0fd7c4d86fd3241fe41cb661ebc5c7321128d59cc295874ab ipmitool_1.8.14.tar.gz
and then unpack it:
tar -xzf ipmitool_1.8.14.tar.gz
and build a 32-bit binary:
cd ipmitool-1.8.14
CC=gcc4 CFLAGS=-m32 LDFLAGS=-static ./configure
make
Note however that despite the -static this will not be a static
binary. This is because ipmitool uses
libtool for linking, and
libtool has its own interpretation of -static: to libtool it
means "do not build dynamic libraries", but it will still dynamically
link executables against "system" libraries. This explains the
problem Ryan Chapman
found
with not getting a static binary. To really get a static binary
you need to tell libtool to use -all-static. But unfortunately
we cannot set that in LDFLAGS, because gcc does not understand
-all-static... and some of the configure tests use gcc directly
for linking, so they will fail :-( The work around is to manually
staticly link the binary again (after building with -static):
cd src
/bin/sh ../libtool --silent --tag=CC --mode=link gcc4 -m32 \
-fno-strict-aliasing -Wreturn-type -all-static \
-o ipmitool.static \
ipmitool.o ipmishell.o ../lib/libipmitool.la plugins/libintf.la
(which is just the Makefile generated command with -all-static in
place of -static, and a new output filename.)
Actually the resulting binary is not completely static. Certain
name resolution related functions like getaddrinfo() and gethostbyname()
will still use dynamic libraries (I assume due to nsswitch). There
will be some static linking warnings about this as a result.
But for simple local IPMI lan configuration it appears to be safe to
ignore these warnings -- the affected code never gets run.
Using the static ipmitool
The static ipmitool can then be copied onto your VMWare ESXi 5.x
server by enabling the ssh server on your ESXi host and then:
scp -p ipmitool.static root@${ESXHOST}:/scratch
ssh root@${ESXHOST}
/scratch/ipmitool.static
For instance:
~ # /scratch/ipmitool.static lan print 1
Set in Progress : Set Complete
Auth Type Support : NONE MD2 MD5 PASSWORD
Auth Type Enable : Callback : MD2 MD5
: User : MD2 MD5
: Operator : MD2 MD5
: Admin : MD2 MD5
: OEM :
IP Address Source : Static Address
IP Address : 10.0.9.17
Subnet Mask : 255.255.255.0
MAC Address : 14:fe:b5:d5:52:b4
SNMP Community String : public
IP Header : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP : 10.0.9.1
Default Gateway MAC : 00:00:00:00:00:00
Backup Gateway IP : 0.0.0.0
Backup Gateway MAC : 00:00:00:00:00:00
802.1q VLAN ID : Disabled
802.1q VLAN Priority : 0
RMCP+ Cipher Suites : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max : aaaaaaaaaaaaaaa
: X=Cipher Suite Unused
: c=CALLBACK
: u=USER
: o=OPERATOR
: a=ADMIN
: O=OEM
~ #
If for some reason you need to set the LAN interface, the usual ipmitool
commands for setting will work, eg:
/scratch/ipmitool.static lan set 1 ipaddr 10.0.9.17
/scratch/ipmitool.static lan set 1 defgw ipaddr 10.0.9.1
and you can do ipmitool lan print 1 to verify that the settings were
updated.
So far only basic LAN settings printing and configuration has been tested. But that is sufficient to get enough up without having to take the ESXi server into Maintenance that the rest can be configured through the iDRAC web interface.
ETA, 2015-07-20: It turns out there is another Dell iDRAC specific
IPMI command: "ipmitool.static delloem lan". For an iDRAC Express it
will show something like:
~ # /scratch/ipmitool.static delloem lan get
Shared LOM : shared with lom1
Failover LOM : None
~ #
And for an iDRAC Enterprise it will show something like:
~ # /scratch/ipmitool.static delloem lan get
dedicated
~ #
You can attempt to change it to dedicated if you have an iDRAC
Enterprise license with:
./ipmitool.static delloem lan set dedicated
but if it fails with an error:
~ # ./ipmitool.static delloem lan set dedicated
FM001 : A required license is missing or expired
~ #
then you either have only an iDRAC Express license, or the iDRAC Enterprise license is not installed properly; using a dedicated port is an iDRAC Enterprise feature.
Finally of note, in Dell speak "shared with lom1" means shared
with LAN-on-Motherboard
1 --
ie, shared with one of the regular LAN ports on the motherboard --
rather than "Lights Out
Management" as that acronym
would mean in other vendors documentation. (Dell of course use "iDRAC"
for what other vendors call "Lights out Management", ie, "LOM".)