Create an ESXi Whitebox ISO with Windows Tools

Friday, March 13th, 2009 | VMWare

This is a short tutorial on how to build a custom ESXi v3.5 Whitebox ISO with Windows Tools (and an Ubuntu-VMImage for VMWare Player) by integrating a new driver (RTL8169 network card) into the Community Driver Pack and integrating this pack into the ISO, too.

See “How to create an ESXi v4 Whitebox ISO with Windows Tools“  for creating an ESXi v4 Whitebox.

Use this tools
- UltraIso $29.95 USD / Free Trial
- Ubuntu Server 8.10 for VMWare Player Free
- Bitvise Tunnelier Free for individual use (or Putty, but Tunnelier is preferred)
- WinImage US$ 30.00 / Free Trial

Let’s prepare by downloading the VMWare ISO, the Community Driver Pack and the new driver.

Download VMware ESXi 3.5 Installable Update 3
Download Community Unified oem.tgz v1.1.0
Download Realtek RTL8169 Driver
Download Realtek RLT8168 Driver! New! oem.tgz package!

1. Extract install.tgz from the VMware ISO with UltraIso

2. Use Tunnelier to log into Ubunto and create a new directory called “esxi” (either with SFTP or by command console with mkdir esxi)

3. Move install.tgz to this esxi directory (use Tunneliers SFTP)
Extract the contents by changing to this directory
cd esxi
tar zxvf install.tgz

and remove the tgz file
rm install.tgz

4. Decompress the VMWare diskimage
cd usr/lib/vmware/installer/
bzip2 -d VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd.bz2

5. Make a new directory for the Community Driver Pack
cd
mkdir oem
cd oem

6. Upload the Driver Pack into this directory and rename it to oem.tgz then extract the contents and remove the file with
tar zxvf oem.tgz
rm oem.tgz

7. Prepare a new directory for the realtek driver
cd
mkdir rtl8169

8. Upload the mymods-0.1.tar into this directory and extract the content with
cd rtl8169
tar zxvf mymods.tgz

9. Then add this driver to the pack
cp mod/r8169.o ../oem/mod
cd ../oem

10. Add the PCIID to the simple.map by using the texteditor nano (no, we are not using the “thing called vi”. Thanks).
cd etc/vmware
nano simple.map

11. Insert the line 10ec:8169 0000:0000 network r8169 at the appropriate place:
...
10df:fd00 0000:0000 storage lpfc_740
10df:fe00 0000:0000 storage lpfc_740
10ec:8169 0000:0000 network r8169
1166:0211 0000:0000 storage ide
1166:0212 0000:0000 storage ide
...

12. Now create the updated oem.tgz file and download it to windows
cd
cd oem
tar cfvz oem.tgz *

13. Now download the cd usr/lib/vmware/installer/VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd file to windows and open it with WinImage (Partition0).

14. Use WinImage to replace the oem.tgz on partition 0 and save.

15. Transfer the dd file to ubuntu /esxi/usr/lib/vmware/installer/ and compress it
cd
cd esxi
cd usr/lib/vmware/installer/
bzip2 VMware-VMvisor-big-3.5.0_Update_3-123629.i386.dd

16. Now recreate the install.tgz file
cd
cd esxi
tar cfvz install.tgz *

17. Download install.tgz to Windows

18. Use Ultraiso to replace install.tgz and oem.tgz (yes, we need to replace this one here, too)

19. Burn ISO and start installing ESXi.

ESXi Tips and Tricks
http://www.vm-help.com/

What’s new
Realtek RTL8168 driver by Jens Jacobsen

Tags: , ,

40 Comments to Create an ESXi Whitebox ISO with Windows Tools

bejker
March 13, 2009

I folow this tutorial but my 2 realtek cards 8139 chip, still not recognized by Esxi:(

Dennis Spreen
March 13, 2009

have you tried pressing ALT-F1 (or F2) to gain access to the console? type “unsupported” (without quotes) and use “lspci -v” to take a look at the PCI ID of the realtek 8139.
The PCI ID is a pair of four digit numbers, the vendor ID and device. You can use ‘lspci -v’ or ‘cat /proc/pci’ to find this information.

It should be 0×8139, so replace step.11 line with
10ec:8139 0000:0000 network r8169

Maybe the same driver works for this model, too. Please report if it worked out or not. thanks!

bejker
March 14, 2009

I mistyped 8139, I meant 8169. Today I found one more Realtek NIC with chip 8168 and I’m going to try this procedure with that NIC. I’ll report the result. tnx.

Cosmix
March 14, 2009

Hello ! In step 3) perhaps the correct command is:
tar zxvf install.tgz
Otherwise you shoud insert another step :
gunzip install.tgz
tar xvf install.tar

Maybe in a future version more hardware will be supported by default !
Best regards !

Dennis Spreen
March 16, 2009

@Cosmix: You’re right. Should be tar zxvf, but works even without it as stated (tar recognizes the gz format). But for clarity I’ll change it, thanks!

DaveP
March 17, 2009

I tried to build R8168 driver – the older r1000 driver from their web site seems to be the closest match.
I removed the debug and devel cflags and managed to install and load the module, /var/config.log shows it loaded and named vmnic0.
But it doesnt show as a Management Network selection – anyone managed to get this driver to work? Its also 1MB which is strange.

Cosmix
March 18, 2009

Hello Dennis ! The new version is already there :
http://www.vm-help.com/esx/esx3i/Custom_oem.tgz/CommunityUnifiedDriverPack_v1.1.0_U3-123629.oem.tgz

Best regards!

Dennis Spreen
March 18, 2009

@Cosmix: Thanks, I’ve updated the tutorial community download.

Peter
March 26, 2009

Hi!
I tried to install an 8168 card but when I booting, the system couldn’t load the r8169 module. I got a ‘VmkCtl’ warning after that the system couldn’t load the vmfs3 module, lvmdriver module, and finally I got a “failed to find HD boot partition” message.
What’s wrong?

Jeremiah C.
March 29, 2009

Looks like the RTL8168 isn’t 100% compatible with the RTL8169, see the ubuntu doc.

Gabriel
April 7, 2009

I have an IBM x3440 ESXi U3 working with 1 NIC (on Board) and I want to add another 8169 chipset NIC.

I dont want install it again (many VMs are working)
Is it possible add the new card without installing ESXi?

Regards

Dennis Spreen
April 8, 2009

@Gabriel: just plug it in. take a look at the configuration page – network adapters

Gabriel
April 8, 2009

Do you have the link of the page you say?

Dennis Spreen
April 8, 2009

run virtual infrastucture client, click on the left on your host (esx.local or something), use right menu “configuration”, Hardware: Network Adpaters. Does it show up? should be vmnic1 then

Gabriel
April 8, 2009

Ah, ok
No the NIC is not showed, and it is not showed in esxcfg-nics -l command in ESXi console.

I see PCI in command lspci -v

[...] for the heads-up on the Planex NIC Create an ESXi Whitebox ISO with Windows Tools | The curse of Dennis D. Spreen KIV first cuz a bit lazy at the moment to do hacking. Intel [...]

Dennis Spreen
April 10, 2009

search /etc/vmware/esx.conf for any mac or vminc entries, remove some of them and restart (there is an article somewhere on the vmware forum, can’t remeber where but there is one).

Dennis Spreen
April 10, 2009

Thanks to Jens Jacobsen there is a RLT8168/8111 driver – see download section on top! Works great.

steve friedland
May 29, 2009

I am not a Linux guru. I have a new laptop with an unsupported nic. What is the dummy version of adding the new nic to ESXI. The install goes great, just does not load nic. lspci – v sees the hardware. Please help, trying to avoid 100hrs of nightmare.

Dennis Spreen
May 29, 2009

steve: which nic do you have?

steve friedland
May 29, 2009

The card is a broadcom. I am reimaging the laptop with the Vista OS to confirm exact type. Dell site had too many options. lspci sees the hardware. I have tried vmkload_mod to try some of the defaults in the /mod folder but keeping getting an error. Is there a possibilty of being able to add the integrated nic? I will have the exact details shortly.

steve friedland
May 29, 2009

Dennis- the card actually shows up as a
“Marvell Yukon 88e8040 pci Ethernet fast controller”.

How would I create a esxi install for this card, or modify the install after?

I appreciate any help you have time for.

Dennis Spreen
May 29, 2009

steve: sorry, i have no informations about marvell drivers. if you are familiar with coding, you may adapt the existing linux driver… but i don’t know of any ready compiled marvell drivers. sorry.

rela
June 28, 2009

Dear Dennis,

your blog post is very nice. It’s the first one of all the posts from the VMware help forum and so on which is very detailed and good explained. I changed some things for my personal server but the solution by including the drivers for a clean installation is perfect.
I’m very pleased for your hard work by publishing this here.
It was the one and only post where it is described how to integrate the drivers completely into an iso without installing on a USB-Key. I think this is a bad solution and not for efficient work.

Thank you very much.

Best regards!

Dennis Spreen
June 29, 2009

thanks rela – and a lot prefer the pxe boot (which is far easier to handle if you need to test your own built system drivers) ;-)

Javier
July 7, 2009

Hi! I have a RTL8139d chiset net card.
http://www.realtek.com.tw/downloads/downloadsView.aspx?Langid=1&PNid=6&PFid=6&Level=5&Conn=4&DownTypeID=3&GetDown=false#2

lspci -v show 10ec:8139.

How I can do a custom oem.tgz to use that card?

Thanks!!
Javier.

Dennis Spreen
July 16, 2009

They claim to have it working:
http://ultimatewhitebox.com/nic

Andrew Roe
July 25, 2009

Would you mind updating this procedure (the driver injection) for ESXi 4.0? I’m entry level linux guy at best and am having a hard time finding the correct locations within the new build.

John Wilson
July 25, 2009

I get the following from the oem.tgz from your link. What have I missed??

[root@dooku oem]# tar zxvf oem.tgz
etc/
etc/vmware/
etc/vmware/simple.map
etc/vmware/simple.map.bak
etc/vmware/support
mod/
mod/3w_9xxx.o
mod/ata_piixsc.o

gzip: stdin: unexpected end of file
tar: Unexpected EOF in archive
tar: Unexpected EOF in archive
tar: Error is not recoverable: exiting now

Dennis Spreen
July 27, 2009

@John: looks like the file is corrupt. try downloading again, if still not working contact me, I’ll send you the file via email.

Santo
August 4, 2009

Hi,

I have a Dell Vostro 410 with a Realtek RTL8168C(P)/8111C(P) NIC and am able to install ESXi 3.5 with this custom oem.tgz

But as soon as I install the vmware tools in my vm image (win2k3 enterprise), the network adapter is behaving flaky:
- IE6 gives “400 Bad Request” when trying to access a webpage (e.g. http://www.google.com)
- Browsing my local network with windows explorer through “My Network Places -> Entire Network -> …” does not work, but accessing a server directly through its ip address (e.g. \\192.168.1.200) does work from time to time, though it’s very very slow and often times out
- ping does work.
- nslookup does work
- tracert does work

Without the vmware tools installed (or by just uninstalling the vmxnet driver from the tools), the network works without any problems but unfortunately the speed is only 10Mbps without those tools

Anyone else experiencing this or any suggestions to solve the issue ?

Note: I prefer ESXi 4 above ESXi 3.5 but unfortunately there is no realtek driver available (yet) for v4 afaik.
Any ideas if/when extra drivers will become available ?

Bart
August 9, 2009

I am trying to follow the above instructions, but are stuck on step 2. Log in to Ubuntu with tunnelier. What IP address to use? I assume 127.0.0.1?
Userid / password user/user?
Whatever I try, I get “no connection could be made because the target machine actively refused it”.
Do I need to start a service in Ubuntu first??

Cory
August 10, 2009

Hello,
Trying to use the R8168 driver from oem.tgz by Jen and having the issues described with it. Are there any updated drivers that DON’T have this issue?
Thanks

Shadrach
December 29, 2009

i followed your steps but it is not working for me not only me most of the friends are facing some problem so could you please upload edited version iso file (realtek nic suppost iso) on rapidshare or megaupload site and give us the link

Luis Guerrero
January 21, 2010

Hi, same comment as Shadrach did.
Im newbie at Linux and at this job i cant spare 1 or 2 hours trying to set it up; jst cant!! there´s to much work to do.
It could save us a lot of time if somebody could help us uploading fixed ISO ESX installer for Realtek.

Thanks a lot for all the job, the step by step guide and efforts to share it with us. Maybe ill spend a couple of overtime hours this weekend.

Dennis Spreen
January 21, 2010

Due to copyright restrictions please do not post a “realtek esxi ISO” link here, thanks (and sorry).

Srinivas Kagitapu
February 7, 2010

Even after injecing the OEM.TZ for 8168, it was throwing the same LVM driver failed. After inspecting a little bit I found some of the lines were mis-aligned in the Simple.map and pci.ids files. The mis-alignement was found exactly where the 8168 reference was provided. When aligned the lines and used the iso again, it worked as it should be.

Thanks
Srinivas Kagitapu

Dennis Spreen
February 10, 2010

thanks for yor hint!

Leave a comment

About

I'm an avid programmer working on a variety of platforms in a variety of languages with a wide technical interest.

QR Code

Search

Categories

Blogroll