How to create an ESXi v4 Whitebox ISO with Windows Tools

Monday, July 27th, 2009 | VMWare

This is a short tutorial on how to build a custom ESXi v4.0 Whitebox ISO with Windows Tools (and an Ubuntu-VMImage for VMWare Player) by integrating a new driver (RTL8169 network card) into the Community Driver Pack and integrating this pack into the ISO, too.

See “Create an ESXi v3.5 Whitebox ISO” for creating an ESXi v3.5 Whitebox.

1. Warning:
You need a 64bit CPU in order to run ESXi v4! Download Processor Check for 64-Bit Compatibility on the VMware site and determine whether your CPU is supported!
Btw: the Intel T2300 CPU is NOT a 64bit CPU…

2. Warning:
There are reports that the RTL driver may not work on all boards because of the new 64-bit portion of the ESXi driver model. Try ESXi 3.5 then. Or wait until a new full supported 64bit driver is made. Check vm-help forum regularly.


ESX Hardware Requirements
VMware ESX 4.0 will only install and run on servers with 64-bit x86 CPUs. Known 64-bit processors:

  • All AMD Opterons support 64 bit.
  • All Intel Xeon 3000/3200, 3100/3300, 5100/5300, 5200/5400, 7100/7300, and 7200/7400 support 64 bit.
  • All Intel Nehalem (no Xeon brand number assigned yet) support 64 bit.

Use this tools
- UltraIso $29.95 USD / Free Trial
- Ubuntu Server 8.10 for VMWare Player Free
- Bitvise Tunnelier Free for individual use (or Putty, but Tunnelier is preferred)
- WinImage US$ 30.00 / Free Trial

Let’s prepare by downloading the VMWare ISO, the Community Driver Pack and the new driver.

Download VMware ESXi 4 Installable
Download Community Unified oem.tgz v1.1.0
Download Realtek RLT8168 Driver oem.tgz package
Download Realtek RTL8169 Driver (if needed, try above first!)

1. Extract image.tgz from the VMware ISO with UltraIso

2. Use Tunnelier to log into Ubunto and create a new directory called “esxi” (either with SFTP or by command console with mkdir esxi)

3. Move image.tgz to this esxi directory (use Tunneliers SFTP)
Extract the contents by changing to this directory
cd esxi
tar zxvf image.tgz

and remove the tgz file
rm image.tgz

4. Decompress the VMWare diskimage
cd usr/lib/vmware/installer/
bzip2 -d VMware-VMvisor-big-171294-x86_64.dd.bz2

Skip step 5-12 if you’re using the Realtek RLT8168 Driver oem.tgz package

5. Make a new directory for the Community Driver Pack
cd
mkdir oem
cd oem

6. Upload the Driver Pack into this directory and rename it to oem.tgz then extract the contents and remove the file with
tar zxvf oem.tgz
rm oem.tgz

7. Prepare a new directory for the realtek driver
cd
mkdir rtl8169

8. Upload the mymods-0.1.tar into this directory and extract the content with
cd rtl8169
tar zxvf mymods.tgz

9. Then add this driver to the pack
cp mod/r8169.o ../oem/mod
cd ../oem

10. Add the PCIID to the simple.map by using the texteditor nano (no, we are not using the “thing called vi”. Thanks).
cd etc/vmware
nano simple.map

11. Insert the line 10ec:8169 0000:0000 network r8169 at the appropriate place:
...
10df:fd00 0000:0000 storage lpfc_740
10df:fe00 0000:0000 storage lpfc_740
10ec:8169 0000:0000 network r8169
1166:0211 0000:0000 storage ide
1166:0212 0000:0000 storage ide
...

12. Now create the updated oem.tgz file and download it to windows
cd
cd oem
tar cfvz oem.tgz *

13. Now download the usr/lib/vmware/installer/VMware-VMvisor-big-171294-x86_64.dd file to windows and open it with WinImage (open partition0).

14. Use WinImage to replace the oem.tgz on partition 0 and save.

15. Transfer the dd file to ubuntu /esxi/usr/lib/vmware/installer/ and compress it
cd
cd esxi
cd usr/lib/vmware/installer/
bzip2 VMware-VMvisor-big-171294-x86_64.dd

16. Now recreate the image.tgz file
cd
cd esxi
tar cfvz image.tgz *

17. Download image.tgz to Windows

18. Use Ultraiso to replace image.tgz and add oem.tgz (yes, as of v4 the file isn’t here by default as it was with v3.5)

19. Burn ISO and start installing ESXi.

ESXi Tips and Tricks
http://www.vm-help.com/


Tags: , ,

31 Comments to How to create an ESXi v4 Whitebox ISO with Windows Tools

alex
August 4, 2009

Is the Realtek RLT8168 Driver oem.tgz package
compatible with esxi 4.0 64bits ???

Thanks

loris
August 4, 2009

I tryed add RLT8168 Driver using http://www.icjacobsen.com/dld/oem.tgz and following your procedure but installation from cd failed for D945GCLF2D / ATOM330

Dennis Spreen
August 5, 2009

See “2. Warning” on top. Some claim to have it working under ESXi 4, some report it does not work. Looks like we need an updated 64 bit driver now. Check the vm-help forums regularly!

Dennis Spreen
August 5, 2009

btw, I’m running the esxi 3.5 server with an Intel T2300 cpu, so no esxi4 for me (on the server with the rtl card), on the other esxi servers I’m using Intel network cards…

Linux_noob08
August 16, 2009

I’m a bit confused regarding step 5-12. The driver pack is a *.oem.tar flie. In linux I just rename things and it still works?

at 8) I assume mymods-0.1.tar is the driver file from Realtek(for example)? Tried it but can’t find any r8169.o or .o-file.

It would be nice if you could make 5-12 a bit more detailed.
/thanks

Dennis Spreen
August 17, 2009

@linux_noob08: no. a *.tar file is a compressed file, an *.o file is a DLL equivalent file, thus just renaming doesn’t work.
just type the commands written above in that order and it should work (well, we need a new 100% compatible rtl 64bit driver, the old one may fail).
If you have any problems just contact me via email and I’ll help you.

Linux_noob08
August 17, 2009

Normally there is a email address hidden somewhere, but sadly I can’t find yours.

Problem is simple:
-Driver package from Realtek is r8168-8.013.00
-The Linux computer I’m working on is not the comp I’ll use the Vmware esxi on. I’ll just create the CD and then boot the esxi installlation on the server, which uses a realtek 8168c/8111c NIC. Checked that twice.
-The Linux comp uses an Intel driver, I just downloaded the Realtekdriver in hope to find the r8168.o file as mentioned in step 9. Only a r8168.h file exist. That is where I’m stranded.

How do I create/make/download the .o version of the 8168-driver for step 9? I tried compiling according to the readme and downloaded missing Kbuild/gcc/make packages. Still no progress. I’m such a noob.

loris
August 20, 2009

I’ll try version i found here then i’ll report
http://www.vm-help.com/forum/viewtopic.php?f=8&t=112

loris
August 20, 2009

Sorry it was an old link.
This is the correct one
http://www.vm-help.com/forum/viewtopic.php?f=17&t=697

loris
August 22, 2009

Install failed on a Atom 330 – Intel D945GCLF2D with r8168

Dennis Spreen
August 23, 2009

@loris: thanks for your effort!

Sacx from RandomBugs
August 25, 2009

If you are using http://www.icjacobsen.com/dld/oem.tgz with an ESXi 4.0 the you should move the pci.ids from /usr/share/hwdata/ to /etc/vmware, if not it will fails with a nice PSOD (Pink Screen Of Death) :)

Regards

Mario Simoes
August 29, 2009

Hello, Denis,

Doesn’t the driver have the be signed for this to work?
Regards,Mario

loris
September 1, 2009

I tryed putting r8168.o file in mod/ directory and pci.ids, simple.map files in etc/vmware/ directory of oem.tgz
Boot failed. I found in log file: Vmkctl Warning Loading module r8168 Failed. “Invalid module name r8168″
What’s wrong?

Oswaldo
September 12, 2009

Hi there, thanks for this guide … one suggestion, you might want to use the windoze version of bzip2 (http://gnuwin32.sourceforge.net/packages/bzip2.htm) to reduce time transfering the files back and forth to linux.
regards

malle
September 14, 2009

thx olswalo. stfp is so slow, using ubuntu live cd to make changes.
suggestion too: winrar can unzip bz2.

im tryin to get esxi run on intel essential. i have a d-link dge-528t which is similar to rtl8169, will hope is works :-D i git the d-link card run with x64 drivers from realtek webseite, but i had to change driver.inf for win server 2003 x64, d-link does not provide x64 server drivers, very bad !

malle
September 14, 2009

HOOOOOOOORAH it works with modified oem.tgz. thanks ! no lvmdriver load error any more.

loris
September 14, 2009

Could you help me getting it working too?

Malle
September 15, 2009

ha sorry git another module error, it seems that r8169 module does not sopprt my dlink vender id ( 1186:4300 ) i put it in simple.map and pci.ids, but it seems that it has to be written into the module too. in 2.6 kernel source it is, but not in the vmware esxi module it seems. so i will buy online a used intel card. sorry

Dennis Spreen
September 21, 2009

malle: thanks for your investigation though!

asoncodi
September 23, 2009

To save transfer times, the tar/compress steps can be performed under Cygwin. http://www.cygwin.com/

kazso
September 30, 2009

This will never work. Basically you just integrate 32-bit drivers (Community Unified oem.tgz, R8168 oem.tgz) into a 64-bit system (ESXi 4). I wonder if anyone tried this out before posting this guide here…

Dennis Spreen
September 30, 2009

You’re somewhat right – see my “2. Warning”. I’ve read that some used the 64 bit drivers and it worked (could’nt verify since I’m not using those cards on an esxi4). The procedure to include a driver is written in here – you only need the 64bit driver – see “Check vm-help forum regularly” above.

Dave McLeod
November 5, 2009

Hi
I tried the steps here and got a checksum error boot failed press any key to continue…

I have a Gigabyte board GA-MA78GM-US2H board with
00:11.0 SATA controller: ATI Technologies Inc SB700/SB800 SATA Controller [AHCI mode]

I installed VMware workstation on Centos 5, thats how I got the SATA controller using lspci.

I then copied the ahci.o file from the Centos 5 and added it according to the steps above.

the iso I created refuse to boot with the error above.

Please help!!

Thank you

joker
December 16, 2009

Hi guys, I’ve succeeded to boot ESXi 4.0 on Intel DG41RQ board (realtek 8111D onboard) using this: http://www.vm-help.com/forum/viewtopic.php?f=12&t=1272

Thanks for this guide!

Bane
December 17, 2009

Just wanted to mention that this whole process can be done on windows by using bzip for windows available at:

http://gnuwin32.sourceforge.net/packages/bzip2.htm

Nime
January 27, 2010

Can anyone make a patch for ESXi 4.0 Update 1 with R8169, there is a free patch maker for Windows at http://download.cnet.com/Patch-Maker/3000-2383_4-10038161.html?tag=mncol

It’s much easy than publishing a how I did article. Thanks.

Zinou
February 12, 2010

Hi all,
I”ve tried to install ESXi4 on Intel core 2 duo E8400 3ghz, 4G RAM, Realtek RTL 8139/810X but always I get error message :
“No compatible network adapter found please consult Hardware Compatibility Guide for supported adapters”

I’ve tried with “Realtek RLT8168 Driver oem.tgz package” given in this site but no result.

Can any one please give me a link where can I find the right driver for my network adapter?

KR,

Zinou.
;

proxyd
February 13, 2010

And I have a similar problem, I will be very grateful for the help.

satish
February 25, 2010

Anyone else seeing the lvmdriver issue? With the above realtek oem.tzg I no longer hit the “No compatible network adapter ..” but now I see

lvmdriver load failure

I’m running on an Asus intel i7 quadcore with

Disk: Hitachi HDT721010SLA360 ATA Device
NIC: Realtek RTL8168C(P)/8111C(P) Family PCI-E Gigabit Ethernet NIC (NDIS 6.0)

Some of the newsgroups say that it is still a nic issue. Any suggestions or leads would be appreciated. I’ve tried with esxi3.5 and lots of permutations and combination.

Please bare in mind I’m relatively new to vm world. Have only brought up one old dell in the past (with hardly any issues).

Thanks,
Satish

mahan
March 9, 2010

i do all the things here but still no compatible network adapter find
please help me
thanks alot

Leave a comment

About

I'm an avid programmer working on a variety of platforms in a variety of languages with a wide technical interest.

QR Code

Search

Categories

Blogroll