Booting VMWare ESXi from USB on VMWare Fusion

A client wanted to be able to boot the VMWare ESXi 6.0.0b installer from a USB drive -- and asked how create a bootable USB drive using OS X. In order to test this, I wanted to be able to boot the created USB drive in VMWare Fusion and test if it booted. This blog post lists a couple of the options that did work (and leaves out several hours of options that did not work).

In the process of testing this I found out:

VMware Fusion does not directly support booting from USB, but apparently Parallels does using EFI boot. VMWare's recommended solution is to image the USB drive into a hard drive image, and boot that -- but that is both slow (imaging the whole USB Drive can take a while if it is large), and does not necessarily prove the USB drive itself is bootable.
The VMWare ESXi 6.0.0b instructions for making a bootable USB drive require a Linux system, because a key step is running syslinux to add boot files onto the USB drive, and copying the syslinux MBR (Master Boot Record) onto the drive -- without those steps it will not boot.
By enabling VMWare Fusion's EFI firmware it is possible to create a USB drive on OS X which will can be booted via the VMWare Fusion EFI internal shell, and might boot on a real system that supported EFI boot. It is fairly clumsy though, at least in VMWare Fusion 6. (And I do not have a bare metal system handy that supports EFI boot to test if the USB drive created only on OS X will boot automatically on an EFI bootable bare metal system.)
The client's system probably also supports the ability to present a virtual CD straight from the original VMWare ISO image, using the iDRAC functionality. Which probably negates the need for a bootable USB drive image anyway. But having spent so long researching options I wanted to write up my notes for future reference.

The official process to make a VMWare ESXi 5.x/6.x bootable USB drive

VMWare supply ISO images for installing VMWare ESXi 5.x and 6.x. They are designed to be written to a CD, but will also work if presented as a Virtual CD -- and that's probably a more common use case these days.

VMWare's ESXi documentation has guides on making ESXi 5.x bootable USB drives and ESX 6.x bootable USB drives. This process requires using syslinux, which needs to be run on a Linux system. VMWare say they have tested syslinux 3.86 and 4.03 -- and I found syslinux 4.05 also worked for me (which is present in Debian Wheezy, and Ubuntu 14.04 LTS).

The process, given a USB drive you are willing to completely overwrite, is approximately:

Download VMWare ESXi 6.0.0b or similar (requires a VMWare Login with access to that product; applying for a trial license should in theory be sufficient). This currently gives you VMware-VMvisor-Installer-201507001-2809209.x86_64.iso which is what is used below.

Partition the USB drive with a single FAT32 partition (type C) covering the whole drive, which is the active partition. This can be done on OS X or Linux. On OS X, use the Disk Utility.app, choose "1 Partition", and "FAT", and it will create a FAT32 partition providing the USB stick is not tiny. To mark it bootable on OS X:

diskutil umount /dev/disk2s1
sudo fdisk -e /dev/disk2
flag 1
update
write
quit

On Linux, the whole process is something like:

sudo fdisk /dev/sdc
p    # print existing partion(s)
d    # delete existing partition(s)
n    # new partition
p    # ... primary partition
1    # ... partition #1
...  # ... accept defaults to use whole drive
t    # change partition type
1    # ... of partition #1        
c    # ... to 0x0C (FAT32)
a    # activate booting of partition
1    # ... number 1
w    # write partition table

to make the partition, and then:

sudo /sbin/mkfs.vfat -F 32 -n NAOS-8GB /dev/sdc1

to put a FAT32 file system on it.

Use syslinux to put the boot files onto the drive. This step requires Linux, as far as I can tell. On Debian Wheezy:
```
sudo apt-get install syslinux
sudo syslinux /dev/sdc1
```
Copy the syslinux MBR onto the drive (needed for syslinux to boot AFAICT). This step could possible be done on OS X (if you got the right file out of the right syslinux version) but is easiest done on the same Linux system as the previous step:
```
sudo sh -c 'cat /usr/lib/syslinux/mbr.bin >/dev/sdc'
```
(The mbr.bin is a 440 byte file, which has just the boot code part of the first sector, so the partition table is not overwritten.)
Mount both the ISO image (loopback mount) and the USB drive, and copy the contents over. On OS X it appears sufficient to do:
```
open ~/Downloads/VMware-VMvisor-Installer-201507001-2809209.x86_64.iso
```
and plug in the USB drive, then select everything and drag and drop it onto the USB Drive.

On Linux, mount the CD image:
```
sudo mount -o loop /var/tmp/VMware-VMvisor-Installer-201507001-2809209.x86_64.iso /cdrom
```
and the USB drive:
```
sudo mount /dev/sdc1 /mnt
```
and then copy everything across:
```
sudo cp -r /cdrom/* /mnt/
```
Rename isolinux.cfg to syslinux.cfg, eg:
```
sudo mv isolinux.cfg syslinux.cfg
```
(isolinux is used to boot a CD; syslinux is used to boot a "regular" hard drive.)

Edit syslinux.cfg and change the "APPEND" line to read:

APPEND -c boot.cfg -p 1

The full contents looks like this:

DEFAULT menu.c32
MENU TITLE ESXi-6.0.0-20150704001-standard Boot Menu
NOHALT 1
PROMPT 0
TIMEOUT 80
LABEL install
  KERNEL mboot.c32
  APPEND -c boot.cfg -p 1
  MENU LABEL ESXi-6.0.0-20150704001-standard ^Installer
LABEL hddboot
  LOCALBOOT 0x80
  MENU LABEL ^Boot from local disk

Assuming you are using syslinux 4.x, overwrite menu.c32 with the version from your syslinux install, as the one supplied by VMWare is for syslinux 3.x and will hang on when booted via syslinux 4.x. To do this:
```
cp /usr/lib/syslinux/menu.c32 /mnt
```
Unmount/eject everything; the USB drive is hopefully all ready to boot.

I think you could do most of the steps on OS X first, and then go do the syslinux steps on a Linux system (syslinux /dev/sdc1, copying the syslinux MBR, copying the syslinux menu.c32) but I have not tested that. I also imagine one could replicate what syslinux /dev/sdc1 does -- eg, which files it copies, but I have not tried to reverse engineer precisely what is required.

Testing USB boot on VMWare Fusion -- Legacy BIOS boot

With some useful hints and much experimentation, I found out that the only way to test legacy BIOS style USB booting with VMWare Fusion is to use the Plop Boot Manager. The Plop Boot Manager is downloadable and usable for free, but is basically "donations appreciated" and "no commercial redistribution". Amongst other features it is able to enable USB booting on systems that do not natively support it, such as VMWare Fusion.

To use the Plop Boot Manager with VMWare Fusion to boot VMWare ESXi 6.0.0 from the USB drive, the easiest solution (per A Virtual Den) is to extract the boot floopy image, and point VMWare Fusion at that:

unzip -j plpbt-5.0.15.zip plpbt-5.0.15/plpbt.img

then in the VM Settings add a Floppy Drive, backed by plpbt.img, and ensure the floppy drive is early in the boot order (it should be by default).

Then start the VM, plug the USB drive in, and use the VMware Fusion hardware management tools to mount your USB drive into the VM. When the Plop Boot Manager menu appears choose "USB" (two down in the menu), and the VMWare ESXi installer should start from the USB drive. (If the boot hangs at that point, the most likely cause is that you've forgotten one or more of the syslinux-related steps listed above -- in particular not running syslinux or not copying the syslinux MBR onto the USB drive are likely to cause the boot to hang.)

Testing USB boot on VMWare Fusion -- EFI boot

It is also possible to boot the VMWare Fusion ESXi 6.0.0b via EFI (again per A Virtual Den). This method makes it possible to skip the syslinux steps described above, and thus the whole process can be done on OS X. Ie, making a single partition with FAT32, marking the partition as bootable, and copying the contents of the ISO image into that FAT32 partition are the key steps. I did also rename isolinux.cfg to syslinux.cfg and edit it, but I am not sure if those steps were needed.

EFI boot is not the default in VMWare Fusion, at least if you choose something like VMWare ESXi 5.x when you are creating the VM image. What seems to work is:

Exit VMWare Fusion
Find the .vmx file related to your test VM
Edit the .vmx file and add a line telling VMWare Fusion to use EFI:
```
firmware = “efi”
```
(at the end is fine.)
Start VMWare Fusion
Start the VM
Plug the USB drive in, and use the VMWare Fusion hardware tools to attach it to VMWare Fusion.
Wait for the EFI boot process to time out and go through to the Boot Menu (saying "No Operating System Found"). (You could try to interrupt it early on with the ESC key -- but when I try it appears not to have recognised the USB drive, so fs0: cannot be selected in the EFI Internal Shell below . So possibly better to wait it out. If it does not find it, you may have to power off the VM, power it on again and try again.)
Choose "Boot Manager", then "EFI Interal Shell" (which is an "Unsupported option")
Enter "fs0:" at the Shell prompt, to change to the removable drive attached earlier.
Enter "cd \efi\boot" to change to the EFI boot directory. "ls" will work in that directory, but note that backslashes are required (not forward slashes -- an interesting mix of OS conventions there!)
Enter "bootx64.efi" to run the 64-bit EFI boot process
"Loading ESXi installer" should appear, followed by various component files.

On a system that did support booting from USB via EFI, it might be possible to choose "USB EFI Device" -- but that has never appeared for me on VMware Fusion 6.x. Possibly some later version of VMWare Fusion will support USB booting directly. (The VMWare Fusion guide to installing OS X 10.7 in a VM suggests that it is possible to get an EFI USB boot option. Choosing "OS X 10.7 64-bit" seems to avoid a long timeout on EFI network boot attempt, but does not appear to give an EFI USB boot option that I can see.)

(It is also possible that configuring the USB device to connect automatically to the VM would both minimise the hassle and improve the reliability of finding the USB drive this way.)

PS: I did experiment a while with making raw disk mapping, eg:

'/Applications/Purchased/VMware Fusion.app/Contents/Library/vmware-rawdiskCreator' \
    create /dev/disk2 1 USB-DRIVE1 ide

and then adding:

ide1:1.present = "TRUE"
ide1:1.fileName = "USB-DRIVE1.vmdk"

to the .vmx file. While that appears to work to map the raw drive (although VMWare Fusion will consider it a "Boot Camp" drive, and ask for your password to authorize direct access on each boot), it appears to only map partitions and thus not allow booting. (Note also that in previous versions this was in /Library/Application Support, but it appears not to be there any longer.)