We have a few of these Apple Xserve machines at work which weren’t doing much, so I thought I’d make better use of them. Naturally, this meant installing Linux on them.
These machines do not have a BIOS (or even any emulation), they use EFI and as such won’t boot the standard Linux install media. I knew that Fedora could boot EFI, so that’s where I started, with Leonidas (version 11). Unfortunately, the install media just wouldn’t work on this device, presumably as it has no BIOS emulation.
To cut a long story short, I had to learn how EFI works in order to get it booting and it wasn’t an easy thing to discover!
Booting EFI
In short, the Apple EFI boots and scans all available drives (including USB drives) for a GPT partitioned device which has a FAT formatted partition on it. Inside this partition it looks for an efi directory.
In that directory must be another from the list of approved vendor named directories, such as redhat or apple. The default is boot and this is what I used.
Inside that boot directory, EFI looks for a binary EFI file, which it can run. In my case, this is a GRUB2 EFI binary. The name of the file must match your architecture, so for me it is called bootia32.efi but if you’re running a 64bit system it should be bootx64.efi.
NOTE: Your Mac must have a 64bit EFI to load the 64bit EFI Linux installer (bootx64.efi)). Else, try choosing the 64bit options when using the 32bit EFI installer (bootia32.efi), but either way your system must have a 64bit CPU.
The GRUB2 EFI binary I am using looks for the standard GRUB config file, which is called grub.cfg. This must also be in the efi/boot/ directory of the USB device. The kernel and initramfs for the net installer should also be in that directory.
I have made a Debian Squeeze, Fedora Constantine and Ubuntu Karmic installer which you can download and extract onto your USB device (sha1sum). You will still need to partition and format your device, so more on how to do that later.
Partition USB Boot Device
So, once we understand how EFI works, next we need to get a bootable GRUB happening in order to install Linux.
Grab a USB memory stick and let’s prepare it. Please know what device it is you’re partitioning!! I’ll use sdz just for example purposes only.
Unmount it
su -c 'umount /dev/sdz1'
Partition and format it
Partition and format it
su -c 'parted -s /dev/sdz mklabel gpt mkpartfs EFI fat32 0% 100% toggle 1 boot'
It should now automatically mount. If not, unplug it and plug it in again, or mount it manually.
Prepare USB Boot Device
Next you can extract my tarball onto the device, ensuring that it maintains the efi/boot/ directory structure on it and not create an extra directories because of the archiver. If it’s not right, then re-arrange things and eject the device.
You should now be able to plug it into your Xserve and turn it on.
Hold down the Option key (Alt key on non-Apple keyboards) until you see the boot option. It should show “EFI” on the usb stick. Boot this and you should see the GRUB menu load. Choose the distro you want to install and away you go! See the “Installation” section below and skip the next few.
Manual preparation – Building GRUB2
Before we can compile GRUB, we need some dependencies. If you’re using Debian or Ubuntu, then run the following:
sudo apt-get build-dep grub2
sudo apt-get install libusb-dev
We need to check out GRUB from CVS and build it for EFI systems. Note, my Xserve is 32bit only, so I’m installing 32bit version. If you’re using 64bit then use a 64bit OS. You can cross compile, see the link above for info if you need to.
Check out source from CVS:
cvs -z3 \
-d:pserver:anonymous@cvs.savannah.gnu.org:/sources/grub \
co grub2
cd grub2
Or get source from Debian (which is more likely to compile):
apt-get source grub2
cd grub2*
Build grub2:
./configure --with-platform=efi
make
Now, make a temporary directory in which to put the GRUB EFI binary.
mkdir -p efi/boot/
Next we need to create the GRUB2 EFI program, and will specify a bunch of modules to include. This was a big stumbling block for me because someone removed the ability to read config files from module “configfile” and put it into “minicmd” requiring “sh”. The documentation I was following was not updated and I couldn’t work out why GRUB2 couldn’t parse my config (typing it in manually at the GRUB2 command prompt worked however). After lots of wasted time I emailed the GRUB list and had an answer within 20 minutes, which was a big relief.
./grub-mkimage -d . -o bootia32.efi part_gpt hfsplus fat ext2 normal sh chain boot configfile minicmd linux loadbios reboot appleldr halt search
sudo cp bootia32.efi efi/boot/
The efi/ directory now just needs to be copied onto your FAT USB stick. Don’t forget to make sure the directory structure is correct, as laid out above.
Now you just need to create a GRUB2 config file, which we’ll do below after getting the kernel and initramfs.
Manual preparation – Linux installers
I am using the netboot installers for each distro. This basically consists of a kernel, initramfs (which includes the installer). To know how to boot these in the GRUB config file, I read the isolinux configs included on the install CDs.
So, download the network installer of your choice in ISO format. Mount loop it and copy the kernel and initramfs onto the efi/boot/ directory on your USB disk.
For example (32bit), Ubuntu, Fedora, Debian Squeeze (note that Fedora does actually have an EFI boot disk which you can dd directly to a USB stick and boot, however this didn’t work on my Xserve but it might be useful to grab the kernel and initramfs from as it’s smaller).
Mount loop ISO:
mkdir /tmp/iso
sudo mount -o loop boot.iso /tmp/iso
Copy the kernel and initramfs from /tmp/iso onto your USB stick. You might need to look at the isolinux.cfg files under /tmp/iso in order to find out which is the kernel. On Debian and Ubuntu the kernel is called linux and the initramfs is called initrd.gz, both in the root directory.
GRUB Config File
Now that you have the GRUB2 EFI binary (bootia32.efi), kernel and initramfs on the USB stick, we need a config file for GRUB to read when it loads.
Here is an example for all four distros that I use in my provided tarball.
set timeout=10
set default=0
menuentry "Debian Squeeze" {
linux /efi/boot/linux-squeeze priority=low vga=normal video=efifb
initrd /efi/boot/initrd-squeeze.gz
}
menuentry "Fedora Leonidas" {
linux /efi/boot/vmlinuz-leonidas nomodeset xdriver=fbdev noselinux
initrd /efi/boot/initrd-leonidas.img
}
menuentry "Ubuntu Jaunty" {
linux /efi/boot/linux-jaunty priority=low vga=normal video=efifb
initrd /efi/boot/initrd-jaunty.gz
}
menuentry "Ubuntu Karmic" {
linux /efi/boot/linux-karmic priority=low vga=normal video=efifb
initrd /efi/boot/initrd-karmic.gz
}
Note that for Fedora I had to turn kernel based mode-setting off, and specify the Xserver driver.
The USB keyboard just wouldn’t work on either Ubuntu, but Debian Squeeze worked fine. I ended up choosing Fedora.
Manual preparation – Install Linux
Once you have all the required files, plug the stick in and turn on the Xserve. Hold down the Option key (Alt key on non-Apple keyboards) until you see the boot option. It should show “EFI” on the usb stick. Boot this and you should see the GRUB menu load where you can boot your network installer of choice.
Installation
When installing, you need to follow the same partition structure as the USB stick. I completely blew away all Apple partitions and am booting only Linux. The drives need to be GPT and to have a small FAT partition at the beginning with the efi/boot/ directory, GRUB2 EFI binary, GRUB config, kernel and initramfs.
There’s probably a better way to do this, I’m sure some distros will support GRUB2 and EFI out of the box soon. In the mean time I install the OS without a boot loader and create the efi/boot/ system manually. This means every time you update your kernel, you need to copy the kernel and initramfs, and update the GRUB config file.
The config file should be the same as the one used on the USB stick, except that the names of the kernel and initramfs will be different. For example:
set timeout=10
set default=0
menuentry "Fedora Leonidas" {
linux /efi/boot/vmlinuz-2.6.28-13-generic nomodeset xdriver=fbdev noselinux root=/dev/sda3 ro
initrd /efi/boot/initrd.img-2.6.28-13-generic
}
Conclusion
These are some rough notes to hopefully help others out there trying similar things. Please let me know how it works for you. I installed Fedora on my Xserve and it’s running quite well!
171 thoughts on “Linux on an Apple Xserve EFI only machine”
@Chris,
Why didn’t you use “fakebios” in your menuentry?
This got me booting without problems into fedora 11 live CD.
Installing without any problems…
(I am running fedora 11 x86_64 on my Xserve1,1)
Give me 2 days and I will post my findings and installation process on my forum.
Hey Martijn,
I did try that with Ubuntu but it still broke. I just took the Fedora grub.conf settings from their efidisk.img which didn’t have it so I didn’t think to add it.. I dunno.. call me crazy 🙂
Thanks for the tip though. Do let me know how your process went!
-c
Hi Chris,
FWIW IA64 is Itanic, X64 is AMD64/EM64T/ia32e.
cheers!
Chris
Hey Chris,
Of course, you’re right.. I’ll fix that, thanks!
-c
Wow. They are some pretty big hoops, although at the end of the day, it is still only an x86 processor.
I’m guessing the reason ordinary Mac users (iMac, MacBook) don’t have to go through these hoops is because tools such as Boot Camp can trigger BIOS emulation?
I’d love to hack on an Intel Mac just for the purposes of playing with EFI some day.
Yeah, the problem is the EFI only system. Other Macs have BIOS emulation which actually has nothing to do with Bootcamp. Bootcamp is just a way to partition the Mac in preparation to install Windows, and creates a driver disk. The actual support for MBR already exists in the firmware of the machine.
-c
Hi, Chris!
I create USB stick as you discribe. But it isn’t shown in boot options.
I will investigate my problem.
Do you know the way to see EFI boot logs or anything that can help to locate issue?
Is your USB stick is shown in Preferences -> Starup Disk as bootable?
How do you think can Mac Mini boots from this flash?
I know, that Mac Mini can boot with Bootcamp, but interesting to know is it boot from such USB?
I find why my XServe doesn’t boot, I have XServe2.1. It has 64-bit arch as opposed to your 32-bit XServe1.1. I’ll try with 64-bit grub.efi
Whoops, I think I forgot a step. You need to make the stick bootable. I’ll fix the howto, but here’s what to do.
When partitioning in parted, before quit, run the following:
toggle 1 bootquit
Don’t forget to copy the files and hold down Option/Alt on boot.
-c
@Chris
My tutorial is done: http://www.4elements.com/blog/install_fedora_on_xserve_part_three/
Booting Fedora 11 form my Xserve1,1. With use of all 3 internal HD’s…
@Chris
Oh yes I forgot to mention. Used 32BIT efi and it runs the X86_64 fedora 11.
Hey Martijn, thanks for the link. I’ll have a read when I get a moment and provide some feedback 🙂
-c
Chris, did you do any special actions to your XServe detect USB flash as bootable?
According parted my USB stick has flag “boot”, but XServe can’t see it on rEFIt stage.
If you’ve partitioned it with gpt and the first partition is formatted with fat, and you’ve created an /efi/boot/ partition there with the bootia32.efi file then it should get detected.
If you haven’t done _all_ of the above, try that.
-c
I read “_all_ of the above” very attentively. Great thanks for your work. I done all as described but I really can’t boot XServe.
I created USB according your “Partition USB Boot Device”. In last post I answered about your actions over XServe.
As I understand – you created bootable USB (as described), put in into XServe, reboot it and in rEFIt screen you see your USB ? I think my XServe can’t see USB devices on loading stage.
Yep, that’s right. If you’ve done all that, then the only other thing I can suggest is that you make sure you have the latest firmware for the Xserve.
The other thing you can try is this Fedora image, which should at least detect:
ftp://mirror.internode.on.net/pub/fedora/linux/releases/11/Fedora/i386/os/images/efidisk.img
You don’t need to partition, it will wipe the entire USB stick (/dev/sdx):
sudo dd if=efidisk.img of=/dev/sdx
-c
Loading directly from fedora efidisk.img is successful.
I use 64-bit efidisk.img because I have XServe 2.1
It becomes more incomprehensibly why XServe can’t detect my FAT formatting GPT partitioned USB-stick.
I know on GPT partitioned disk there EFI system partition, as I see, you didn’t advice create this on USB. Or I understand something wrong?
OK, then I’m not sure why. Did you try Martijn’s instructions above?
Also, if the efidisk.img works, then you can just replace the files on it with the ones in my instructions above. If your machine is 64bit, maybe that’s why it’s not detecting. You need to use a 64bit build of grub and your efi should be called bootx64.img as per instructions.. but I’m just guessing here. Let me know how you go!
-c
Chris, did you try extract efidisk.img?
According anaconda sources (mk-images.efi) it should be fat32 image, but mac,linux or windows can’t detect correct filesystem on efidisk.img
No, I didn’t extract it, just write it. It is an image of a GPT partitioned fat32 formatted device. Windows won’t see it. Mac won’t see it. Only Linux is smart enough to talk to it. Plugging it into your Linux machine should bring it up like a normal device, although your memory stick will only be about 25MB.
-c
@Chris
I have some questions…
– Does your installation detect all hardware?
– If you get errors, what kind?
– what’s your overall impression of the speed?
– can you test “hdparm -tT /dev/hda” (without the “”) hda is your disk. don’t know what your HD name is 🙂
Regards,
Martijn
@Martijn,
– Does your installation detect all hardware?
Pretty sure that it does.. I mean everything I’ve been using so far has worked! Anything in particular?
– If you get errors, what kind?
The only annoying thing is the resolution which is stuck at something non-standard like 1320×880. On the Dell monitor I’m using it constantly prints a big black box in the middle saying that it’s an unsupported res. Makes it hard to do things, but other than that I don’t think I have any other errors.
– what’s your overall impression of the speed?
It seems very good. Even kernel based mode setting appears to be turned on (how I’m not sure cause I didn’t think it’s a supported card, plus I’m running in Vesa mode) and it pauses for a minute or so just before the GDM login screen. After that it works well. I’m running 3 drives in RAID5. All 4 CPU cores seem to work well and I haven’t noticed any lag or high CPU usage. RAM usage is sitting low at around 700MB in GNOME.
– can you test “hdparm -tT /dev/hda”
Sure, I might have to get back to you on that one, it’s in the server room test rack 🙂
-c
hdparm /dev/sda showed about 3500MB/sec cached read and 170MB buffered read as far as I recall. Unfortunately I crashed the box and suffered ext4 corruption.. rebuild time tomorrow, with ext3!
-c
@Chris
Thanks for the info.
An other question:
I also have the xserve1,1, using the 32Bit efi. But the system is 32 bit and 64 bit
I tried your info, but when I startup from the usb and pick my x86_64 boot I get only errors. (your 32 bit install works fine)
Did you complile your own vmlinuz-leonidas and initrd-leonidas.img?
If so could you compile a 64 bit version?
@Martijn,
Hey, no unfortunately I didn’t try 64bit and I just used the default leonidas images from their efidisk.img file.
Perhaps you also need a 64bit grub.efi?
-c
@Chris
Hmm, No the 64bit grub.efi won’t work while xserve1,1 only uses 32bit efi.
So 32 bit efi and 64 bit processor etc.
(if you use an intel xserve1,1 as well you can also install 64bit fedora OS)
Why use a 32bit if you can run 64bit?
@Martijn,
Because I thought that 32bit would be less work considering I was wandering into new territory, I only have 2GB RAM and it’s not used for number crunching 🙂
-c
@Chris
Found the solution to install 64Bit OS on your 32bit efi installation story.
edit:
menuentry “Fedora Leonidas” {
fakebios
linux /efi/boot/vmlinuz nomodeset xdriver=fbdev noselinux
initrd /efi/boot/initrd.img
}
vmlinuz and initrd.img copied from 86_64 fedora disk.
startup from usb and do the netinstall.
Works like a charm 🙂 Do not use the ext4 (only ext3)
Right now I am testing if your workaround will give me a faster/stable fedora compared to my workaround.
Will Let you know.
I will add you as a link. Very useful information. If some of you is interested they can also post in my forums.
Regards,
Jeroen
Fascinating research project Chris but I gotta ask, why not just use OS X on them like Jobs intended? You’ve got all the GNU tools and can get tons more OSS via http://www.macports.org/. What can you do on Linux that you can’t on OS X?
Hey Jim,
Thanks for the post. Yes I could do that, but Linux is so much easier 🙂 I can set up a system which just has the exact tools that I want and install things in no time at all. Also, this network is completely disconnected from the Internet, so I need a local repository for installing everything that I want, including updates. Linux fits the bill perfectly.
As for OS X itself, it’s a great operating system but the tools just aren’t the same, plus it comes with so much cruft. MacPorts is just messy, you get the Apple programs, then the MacPorts stuff in different locations. Everything’s all over the place. I also like access to virtual file systems /proc and being able to install and remove kernel modules, etc. I can make the system exactly what I want and have it all work seamlessly.
Linux is just far more flexible and powerful than OS X, but that’s just my opinion 🙂
-c
Hi,
We are trying to boot an XServe3,1 with double Xeon 64.
We have some problems booting with the framebuffer because it don´t find any valid address for it. We get booting with fedora, but vga doesn´t display correctly. We are waiting for a vga minidisplayport adapter but we are continuing trying with diferent options.
Very good work!!
Regards
Hi Sir Chris,
Actually partially I make it happen to boot Xserve on Fedora 11 Live CD.But now I am trying to install the Fedora 11 distro on Xserve yet trying to tweak this grub.cfg ;Can you do the help?
I followed this settings..
####################################################################
# grub.cfg pxw 20090623
timeout=20
default=0
set F1=ctrl-x
set F2=ctrl-c
set color_normal=yellow/blue
menuentry “Fedora-11-x86_64-Live CD boot” {
fakebios
root=cd0
linux /isolinux/vmlinuz0 root=CDLABEL=Fedora-11-x86_64-Live rootfstype=auto ro liveimg rhgb single acpi=force irqpoll video=efifb nomodeset
initrd /isolinux/initrd0.img
}
menuentry “REBOOT” {
reboot
}
#############################################################
Thanks
Christopher
What’s happening when you boot? Does grub2 load the menu list at all?
-c
Hello Chris,
and interesting (and frightening!) article.
I’m trying to install Fedora 12 as the sole O/S on my Macbook Air. I thought that as I can boot the F12 DVD and install it Linux from it, I wouldn’t have any problems, but it doesn’t boot.
The first time, I left /dev/sda1 (the EFI partition) alone, and installed the boot loader on /dev/sda2 (mounted as /boot).
On my second attempt, I deleted the EFI system partition, and installed the boot loader on /dev/sda. That didn’t work either.
Can you suggest an easy way of installing F12, or do i have to go through all this trouble of preparing a USB stick?
Hey Colin,
Do you want to dual boot your Macbook Air, or are you trying to do a Linux only build? I’d recommend doing a dual boot for simplicity’s sake, simply because you can install rEFIt under OSX to give you a nice pretty menu on boot up.
If you want a Linux only boot then maybe (maybe) this will work. Boot the OS X disk (holding down the C key) and use the partition manager to initialise the disk. This will create an EFI Fat32 partition as well as the HFS+ partition for Mac. Quit the Installer and switch to your Fedora 12 disk (holding down the C key again). Boot the Fedora installer and when it comes time to partition your drive, tell it to “use entire drive”.
I’m guessing that it will detect the existing EFI structure, and will therefore set up its own EFI structure when it installs. Fedora can boot directly from EFI..
Give that a shot anyhow and let me know how you went!
Installing the boot loader to /dev/sda simply won’t work on an EFI system because that’s not how it boots. It is how a MBR system boots however, using the first 446 bytes on the drive.
Cheers,
Chris
No, I’m not willing to try dual-boot, as I can’t afford the disk space.
So I tried your suggestion. In this case the installation program chooses to mount /boot on /dev/sda1, where it suggests installing the boot loader, and uses the rest of the disk (the HFS+ partition) as an LVM. This sounds sensible, but I get the same result as before. 🙁
Hmm… ok. I might have to have a play with it myself and see what I can work out.
The only other suggestion that I have at the moment is to boot to the Fedora installer and switch to terminal 2 and create a blank GPT table on it.
parted /dev/sda
mklabel gpt
This way the installer should see just a blank GPT drive, and might set up EFI properly. The bootloader SHOULD go to /dev/sda1.
Mount this in terminal 2 before rebooting and look at the structure. It should probably have an efi directory, with redhat under that with bootloader and kernel, etc.
The other thing I’m thinking is that the Mac EFI doesn’t recognise the efi/redhat structure. You could try holding down the ALT key and see if you can boot it that way, or rename the efi/redhat directory and files to the generic ones I use above.
-c
Actually I now have it working. Peter Danenberg, on the Fedora users list, suggested using the gptsync utility from the refit CD might be all that was left to do. And indeed it was.
OK, so that means that the mac is actually booting your machine in MBR emulation mode. I should have thought do suggest that. The Xservers don’t have MBR emulation, hence they have to be EFI only.
Under your Fedora install, if you install anaconda you’ll get the gptsync command. Interesting that F12 didn’t run that for you automatically..
Glad it’s working, sorry I wasn’t any help.
-c
Hi Chris ! Thanks for the guide ! 😀
I’ve managed to install Ubuntu Karmic (v9.10) on 2 Xserve now. It’s a great relief 😀
However I had to change a few stuff in your efi-grub.tar.bz2 to get it to work.
I’ve uploaded the new version to : http://rimbalinux.com/files/efi-grub-v2.tar.bz2
Nothing big, just minor changes – but essential. I’ll list them :
(1) Changed Karmic’s initrd & kernel to the newest one : On the Karmic’s installer, I kept on getting message that the netinstaller failed to find & install the same version of kernel from the repository.
So I replaced karmic’s initrd & kernel file with the newest one.
Then it’s able to find the same version of its kernel in the repo, and installation was able to continue until the end.
(2) /efi/boot/sample-configs directory : here you can find 2 files :
(2)(a) grub.cfg–harddisk : this is a sample file that you can put as /efi/boot/grub.cfg in your server’s hard disk (once the installation is done)
(2)(b) xorg.conf : this is the correct config file to get the desktop up & running. Copy it to /etc/X11/
Also I assume the following as the harddisk’s partition layout :
(1) you MUST create a small partition in the beginning of your first hard disk, for the EFI files.
Mine is only 250 MB big, and the contents are exactly the same as this efi-grub-v2.tar.bz2 (don’t forget to replace the grub.cfg with grub.cfg–harddisk)
(2) The second partition is the root (/) partition. If not, then you’ll need to adjust /efi/boot/grub.cfg’s file accordingly.
That’s it ! Now my Xserve are running Karmic 😀
Thanks again Chris !
Also I’d like to acknowledge the assistance from the postings on this thread : http://ubuntuforums.org/showthread.php?t=1288738&page=2
The discussions there enabled me to construct the right grub.cfg & xorg.conf file for my Xserver 1,1
Thank you all.
Great, thanks 🙂
-c
I have released an updated tarball, which includes Fedora 12 (Constantine), updated Ubuntu 9.10 (Karmic) installers and supports 64bit.
-c