TL;DR

You can use Docker to build a staticly linked ipmitool that will run on VMWare ESXi 5.x to confgure a Dell iDRAC's network settings. Read on for details of how this works.

Motivation

Imagine that you have a set of Dell servers running VMWare ESXi colocated in a data centre. And that you are concerned that you might need to interact with the servers before they have fully booted -- perhaps when they are not booted -- so that ssh and the VMWare management interfaces cannot be used. In those instances it would be very useful to be able to interact with the console of the remote machines, and at least reassure yourself that they are still booting.

If your Dell Servers were bought with Dell iDRAC support, then you have the hardware you need -- you just need to cable it up to some out of band network (ideally via some independent Internet connection so you do not rely on your own routers having booted), and configure the iDRAC network interface so you can reach it via that out of band network.

The cabling part is fairly easy -- give or take a trip to the data centre. But by default the configuration of the iDRAC can be done on the server only by interrupting the boot process at the appropriate point. Which would mean rebooting the ESXi servers. The production ESXi servers. Yes it is a cluster, but that is a lot of shuffling VMs around to free up one host at a time so it can be rebooted, which is a lot of time in the data centre.

On Linux and Microsoft Windows, Dell provide a racadm tool which can be used to configure basic features of the Dell iDRAC -- enough that you can then get in via the web interface and configure the rest. The only trick is making sure that you use the correct version for your iDRAC card -- inexplicably Dell appear to have not only changing the configuration interface between versions but also provided separate binaries for each version (rather than auto-detecting the installed card and supporting all the protocols in one program). With the wrong version for your iDRAC card, you will be very confused why it does not seem to work.

Unfortuantely there is no VMWare ESXi version of the local racadm (at least that I have been able to find). And the remote racadm needs the IP address and username/password of the iDRAC to talk to... which brings us back to the first problem.

But conveniently the Dell iDRAC is based on IPMI 2.0, which means that IPMI tools can be used to access some of the functionality. (This seems to have been true since the Dell DRAC 5; there is even a Dell guide to using ipmitool with the older PowerEdge servers.)

ipmitool

On Linux, one of the standard tools for interacting with the IPMI (locally and remotely) is ipmitool (which unfortunately seems to still be hosted on the increasingly untrustworthy SourceForge -- but source tarballs are also available via major Linux vendors). Ryan Chapman discovered that a "statically" linked Linux ipmitool would run on VMWare ESXi 4.1, and John found this still works on VMWare ESXi 5.1, as did Mitchell Turner.

The main requirements for using a Linux ipmitool under VMWare ESXi seem to be (a) ensuring it is built on a Linux release that is approximately old enough to match the libc and kernel versions of the VMWare ESXi host, (b) buliding a 32-bit binary, and (c) making sure it is statically linked. (Possibly either matching libc versions or statically linked would be sufficient, but aiming for both maximises the chances of it working.)

On VMWare ESXi 5.0 (still in use due to being a well stabilised combination of software and drivers, pending a jump to ESXi 6.x), the libc there was:

~# /lib/libc.so.6
GNU C Library vmware release version 2.5, by Roland McGrath et al.
[...]
Compiled by GNU CC version 4.1.2 20070626 (Red Hat 4.1.2-14).
Compiled on a Linux 2.6.9 system on 2015-01-29.

compiled with a compiler and on a kernel version which are roughly equivalent to CentOS 4, of which the last release was CentOS 4.9, and support ended about three years ago.

32-bit CentOS 4.9 build environment

So we wanted a 32-bit CentOS 4.9 build environment. Since I did not fancy doing a full install of an ancient Linux system (into a VM) just to build a single tool I went looking for a more convenient answer. It turns out that while the official CentOS Docker images are only 64-bit (x86_64) and only start with CentOS 5, Brian Lalor has produced an unofficial Docker build script to make a base CentOS 4.9 image. Unfortuantely that too was 64-bit (x86_64) but with some simple tweaks it is possible to get it to build a 32-bit (i386) CentOS 4.9 base image:

git clone https://github.com/blalor/docker-centos4-base.git
cd docker-centos4-base
vi build.sh       # Change arch from x86_64 to i386, comment out EPEL
docker run --privileged -i -t -v $PWD:/srv centos:centos6 /srv/build.sh i386
docker build -t centos4_i386 .

The docker run produces a centos49.tar.gz chroot tarball (a bit like debootstrap on Debian), and the docker build builds a Docker image with just that tarball unpacked inside it. (My docker-centos4-base build.sh script takes the architecture as a command line parameter; if I get a chance to make that a bit less hacky I will try to push it back upstream.)

CentOS 4.9 build environment fixups

From there, it just a matter of fixing up the 32-bit CentOS environment to force it to install 32-bit binaries (by default it will install binaries for $basearch which will be 64-bit), and then installing the minimal ipmitool build dependencies. To do this by hand:

docker run -it centos4_i386 /bin/bash
cat /etc/yum.repos.d/CentOS-Base.repo       | 
    sed 's/\$basearch/i386/; s/\]/-i386]/;' |
    tee /etc/yum.repos.d/CentOS-i386.repo
sed -i '/protect=1/a enabled=0' /etc/yum.repos.d/CentOS-Base.repo

(these examples chosen to avoid IO redirection as the angle brackets conflict with the HTML markup :-( ). And then it is necesssary to manually install the CentOS 4 yum key before installing packages:

curl -L -o /tmp/RPM-GPG-KEY-centos4 http://mirror.centos.org/centos/RPM-GPG-KEY-centos4
sha256sum /tmp/RPM-GPG-KEY-centos4
# Should be:
# 57821109aeb5f27805b80be03e31b1d978f3b7c7464c684169b1d341ff4d7021  /tmp/RPM-GPG-KEY-centos4
rpm --import /tmp/RPM-GPG-KEY-centos4 

Building static ipmitool

NOTE: If you have Docker running, you can can use Docker to build a staticly linked ipmitool that will run on VMWare ESXi 5.x automatically. The Dockerfile automates the build environment tidying up above and all the build steps below. The build steps below are listed for completeness in case you want to do it by hand in some other environment.

Building ipmitool is fairly simple, thanks to it having very few dependencies. It just needs gcc4 and make:

yum -y install gcc4 make

First download the source archive, eg:

curl -L -o ipmitool_1.8.14.tar.gz http://http.debian.net/debian/pool/main/i/ipmitool/ipmitool_1.8.14.orig.tar.gz

(this one from Debian's archive, which I trust more than Sourceforge now), and then verify the checksum:

sha256sum ipmitool_1.8.14.tar.gz
# Should be:
# fe9bce4587f83fe0fd7c4d86fd3241fe41cb661ebc5c7321128d59cc295874ab  ipmitool_1.8.14.tar.gz

and then unpack it:

tar -xzf ipmitool_1.8.14.tar.gz

and build a 32-bit binary:

cd ipmitool-1.8.14
CC=gcc4 CFLAGS=-m32 LDFLAGS=-static ./configure
make

Note however that despite the -static this will not be a static binary. This is because ipmitool uses libtool for linking, and libtool has its own interpretation of -static: to libtool it means "do not build dynamic libraries", but it will still dynamically link executables against "system" libraries. This explains the problem Ryan Chapman found with not getting a static binary. To really get a static binary you need to tell libtool to use -all-static. But unfortunately we cannot set that in LDFLAGS, because gcc does not understand -all-static... and some of the configure tests use gcc directly for linking, so they will fail :-( The work around is to manually staticly link the binary again (after building with -static):

cd src
/bin/sh ../libtool --silent --tag=CC --mode=link gcc4 -m32 \
    -fno-strict-aliasing -Wreturn-type -all-static         \
    -o ipmitool.static                                     \
       ipmitool.o ipmishell.o ../lib/libipmitool.la plugins/libintf.la

(which is just the Makefile generated command with -all-static in place of -static, and a new output filename.)

Actually the resulting binary is not completely static. Certain name resolution related functions like getaddrinfo() and gethostbyname() will still use dynamic libraries (I assume due to nsswitch). There will be some static linking warnings about this as a result. But for simple local IPMI lan configuration it appears to be safe to ignore these warnings -- the affected code never gets run.

Using the static ipmitool

The static ipmitool can then be copied onto your VMWare ESXi 5.x server by enabling the ssh server on your ESXi host and then:

scp -p ipmitool.static root@${ESXHOST}:/scratch
ssh root@${ESXHOST}
/scratch/ipmitool.static

For instance:

~ # /scratch/ipmitool.static lan print 1
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD
Auth Type Enable        : Callback : MD2 MD5
                        : User     : MD2 MD5
                        : Operator : MD2 MD5
                        : Admin    : MD2 MD5
                        : OEM      :
IP Address Source       : Static Address
IP Address              : 10.0.9.17
Subnet Mask             : 255.255.255.0
MAC Address             : 14:fe:b5:d5:52:b4
SNMP Community String   : public
IP Header               : TTL=0x40 Flags=0x40 Precedence=0x00 TOS=0x10
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : 10.0.9.1
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14
Cipher Suite Priv Max   : aaaaaaaaaaaaaaa
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
~ #

If for some reason you need to set the LAN interface, the usual ipmitool commands for setting will work, eg:

/scratch/ipmitool.static lan set 1 ipaddr 10.0.9.17
/scratch/ipmitool.static lan set 1 defgw ipaddr 10.0.9.1

and you can do ipmitool lan print 1 to verify that the settings were updated.

So far only basic LAN settings printing and configuration has been tested. But that is sufficient to get enough up without having to take the ESXi server into Maintenance that the rest can be configured through the iDRAC web interface.

ETA, 2015-07-20: It turns out there is another Dell iDRAC specific IPMI command: "ipmitool.static delloem lan". For an iDRAC Express it will show something like:

~ # /scratch/ipmitool.static delloem lan get
Shared LOM   :  shared with lom1
Failover LOM :  None
~ #

And for an iDRAC Enterprise it will show something like:

~ # /scratch/ipmitool.static delloem lan get
dedicated
~ #

You can attempt to change it to dedicated if you have an iDRAC Enterprise license with:

./ipmitool.static delloem lan set dedicated 

but if it fails with an error:

~ # ./ipmitool.static delloem lan set dedicated 
FM001 : A required license is missing or expired
~ # 

then you either have only an iDRAC Express license, or the iDRAC Enterprise license is not installed properly; using a dedicated port is an iDRAC Enterprise feature.

Finally of note, in Dell speak "shared with lom1" means shared with LAN-on-Motherboard 1 -- ie, shared with one of the regular LAN ports on the motherboard -- rather than "Lights Out Management" as that acronym would mean in other vendors documentation. (Dell of course use "iDRAC" for what other vendors call "Lights out Management", ie, "LOM".)