Linux on Mac Pro with multiple drives

30 thoughts on “Linux on Mac Pro with multiple drives”

1. What kind of MacPro is it ?

2. Hey Jeroen,

   Looks like it’s a:
   “MacPro 1.1, Dual-Core Intel Xeon 3GHz, Boot ROM Version MP11.005D.800, SMC Version 1.7f10.”

   Cheers,
   Chris

3. Pingback: Does your Mac have 64bit EFI? at the blog of Chris
4. Pingback: Linux on Mac Pro with multiple drives, for real this time at the blog of Chris

5. Hi Chris,
   Fedora 13 and 14 seemed to install well for me on a secondary drive and Ubuntu even installs with only bootcamp.

   If you have any experience with RHEL please let me know.
   I’ve completely reformatted both my Hard drives out of my Quad core Mac pro stirpped them of everything even the Mac OSx. Still RHEL will install but the machine ignores it completely claiming there is no operating system installed !!

   Take care
   El

6. Hi El,

   Thanks for the update. So you’re not using rEFIt, just bootcamp? Is this an EFI only install, or are you using MBR emulation?

   -c

7. Hi Chris,

   So I haven’t tried your approach with Fedora 14 (and RHEL 6) yet, but will shortly.

   Oddly, I have been able to install Ubuntu 10.10 on a second drive in my Mac Pro without resorting to additional tools, but Fedora and kin… not so much.

   Basically, on my second drive, I create 2 partitions (one for Linux, and one for swap) with diskutil under Mac OS X. I then boot from a Ubuntu LiveCD and mke2fs/mkswap. Then install (selecting manual partitioning) and it works fine. I’ve tried this approach with both RHEL 6 and Fedora 14 and it fails. The disk does not show up as a boot option, and using various tools it’s apparent that the MBR/GPT are out of sync or corrupt.

   I guess my question is, what is Ubuntu doing that Fedora isn’t (or vice versa)?

   I have a 2010 12-core Mac Pro, and the first drive contains only Mac OS X 10.6.

   Thanks!

   -le

8. Hi Larry,

   Good question, I’m not sure. Perhaps you can compare the partition table between a Fedora and Ubuntu install? My guess is that Fedora knows it’s GPT, and so installs appropriately (grub to /boot) not MBR. Ubuntu is probably not that clever, which is having the beneficial side-effect of working 🙂 It would be interesting to see where Ubuntu installs the boot loader. If you synchronised the partitions on the Ubuntu install, does it still boot?

   -c

9. Hi Chris,

   I’ll post output from various utilities for both Ubunto & Fedora in the next day or two, and try syncing a Ubuntu install and see what happens. My setup is probably a little unusual in that the first two disks (where I have OS X installed) are a RAID mirror… it seems to confuse the rEFIt partition analysis tool sometimes, although oddly, not always. I believe when I first tried installing Ubuntu I chose the Linux partition (/dev/sdc2) to install the boot loader, and that didn’t work. I then tried /dev/sdc and that worked. I’ve tried both /dev/sdc and /dev/sdc2 with Fedora/RHEL and neither works.

   Oh, btw, while I think of it, last night I tried to boot the Mac Pro from efidisk.img (Fedora 14, 64 bit version) on a USB stick, and it hangs at the end of the countdown. Gets further on my 2007 MacBook Pro, but still doesn’t completely boot.

   -le

10. Hmm.. maybe it can’t find the disk to boot. Did you dd it correctly?

    -c

11. Well, I tried the dd a couple times… same thing happened both times. Basically:

    dd if=efidisk.img of=/dev/disk5

    On the MacBook Pro the USB stick activity light keeps flashing after the Fedora countdown (and as I say seems to start booting up, but ultimately fails), on the Mac Pro it stops flashing and nothing else happens.

    -le

12. OK, that is probably right.. I wonder if you can manually mount the stick and add “nomodeset” to the kernel line to see what’s going on?

    You don’t actually need to use EFI only on a Mac/Macbook Pro though, because they have a fake MBR built in. It’s only the XServes that have EFI only.

    Theoretically 🙂

    -c

13. Hi Chris,

    Here’s the diskutil partition info for the (working) Kubuntu 10.10 install:

    /dev/disk0
    #: TYPE NAME SIZE IDENTIFIER
    0: GUID_partition_scheme *2.0 TB disk0
    1: EFI 209.7 MB disk0s1
    2: Microsoft Basic Data 2.0 TB disk0s2
    3: Linux Swap 31.7 GB disk0s3

    And the output of gptsync (I didn’t actually change anything with it):

    Current GPT partition table:
    # Start LBA End LBA Type
    1 40 409639 EFI System (FAT)
    2 411648 3845201919 Basic Data
    3 3845201920 3907028991 Linux Swap

    Current MBR partition table:
    # A Start LBA End LBA Type
    1 1 409639 ee EFI Protective
    2 * 411648 3845201919 83 Linux
    3 3845201920 3907028991 82 Linux swap / Solaris

    the rEFIt partition inspector shows the same and adds:

    MBR contents:
    Boot Code: GRUB

    Partition at LBA 40:
    Boot Code: None (Non-system disk message)
    File System: FAT32
    Listed in GPT as partition 1, type EFI System (FAT)

    Partition at LBA 411648:
    Boot Code: None
    File System: ext3
    Listed in GPT as partition 2, type Basic Data
    Listed in MBR as partition 2, type 83 Linux, active

    Partition at LBA 3845201920:
    Boot Code: None
    File System: Unknown
    Listed in GPT as partition 3, type Linux Swap
    Listed in MBR as partition 3, type 82 Linux swap / Solaris

    For the (non-working) RHEL6 install, diskutil shows:

    /dev/disk1
    #: TYPE NAME SIZE IDENTIFIER
    0: GUID_partition_scheme *2.0 TB disk1
    1: EFI 209.7 MB disk1s1
    2: EFI 2.0 TB disk1s2
    3: Microsoft Basic Data 31.7 GB disk1s3

    gptsync shows:

    Current GPT partition table:
    # Start LBA End LBA Type
    1 40 409639 EFI System (FAT)
    2 411648 3845201919 EFI System (FAT)
    3 3845201920 3907028991 Basic Data

    Current MBR partition table:
    # A Start LBA End LBA Type
    1 1 3907029167 ee EFI Protective

    it proposes the following new MBR:

    # A Start LBA End LBA Type
    1 1 411647 ee EFI Protective
    2 411648 3845201919 83 Linux
    3 3845201920 3907028991 0b FAT32 (CHS)
    4 3907028992 3907029167 ee EFI Protective

    which still seems wrong… not sure where it gets that 0b partition type. I think on one iteration I actually let gptsync do its thing, but with no good result.

    The rEFIt partition inspector doesn’t recognize the Linux disk exists, it displays info on the first RAID disk containing OS X.

    For the efidisk.img boot off USB, I’ll see if I can figure out how to add that nomodeset option.

    -le

14. This is where it gets 0b from:
    EFI 2.0 TB disk1s2

    It’s a FAT32 partition, but it’s meant to be ext3.. maybe make sure this is getting formatted as ext3 during install?

    rEFIt only works on the first disk, which is why you aren’t seeing it there..

    Does Ubuntu boot if it’s not on the first disk?

    -c

15. Hi Chris,

    Yep, the Kubuntu install was in the 3rd bay, and it booted fine. The Linux disk identifier changes even though the physical disk is always in bay 3… it can be anything from disk0 to disk3, and changes across reboots. Although, I’m thinking now that when the GPT and MBR are properly synced, and Kubuntu is bootable, it always shows as disk0.

    Prior to install, I manually formatted the main Linux partition as ext4 (also tried ext3 I think) and then told RHEL6/Fedora 14 not to format it during the install (the same procedure I followed for Kubuntu).

    -le

16. Hmm, OK, then the EFI partition type is probably wrong. Either way, EFI thinks it’s a fat32 partition, so the MBR table is wrong.

    -c

17. Yep, I think the MBR is bad… although I believe I tried gptsyncing the MBR (after a RHEL6 install) and the disk still didn’t show as a boot alternative. I believe the gptsynced disk MBR was close, but not identical to the working Kubuntu MBR.

    By the way, I just tried a slightly different way of doing the Kubuntu install, I still partitioned the disk with diskutil, but let the Kubuntu installer format the Linux partition instead of manually formatting it myself beforehand. That actually still resulted in a bootable disk, but the disk oddly now shows up as having a FDisk_partition_scheme (no GPT) in diskutil… but as I say it boots up fine into Kubuntu. I’m tempted to use gdisk to add a GPT based on the MBR, but is that really needed?

    I think I’m going to try installing RHEL6 over the working Kubuntu install and not have it install a boot loader or format things, and see what happens…

    The basic problem is I don’t really understand this hybrid GPT/MBR setup not to mention boot loaders. If I did I probably would be able to use some combo of gptsync & gdisk to set things right, but I’m just groping in the dark.

    -le

18. Well I just realized the FDisk_partition_scheme oddity was left over from my screwing with the disk earlier with gdisk & gptsync trying to get RHEL6 to boot. I assumed diskutil would wipe everything out when I repartitioned, but apparently not. It took the bootcamp assistant to get the GUID_partition_scheme back. Wow…

    Anyway, going to reinstall Kubuntu, and then follow it with installing RHEL6 and not format or install boot loader.

    -le

19. So your drive needs some kind of partition table. Usually PCs use the msdos partition table (sometimes called mbr table), but that has lots of limitations (like max 4 partitions, max partition size of 2TB).

    There’s a more modern partition table, GPT, which doesn’t have these limitations. That’s what Macs use. Macs also don’t have a BIOS, they have an EFI.

    However, the EFI in Macs have a “fake” MBR table too, so they can really pretend to be a BIOS and boot a MBR partitioned drive (except for the Xserves, which do not).

    The master boot record is the first 446 bytes of a drive, and where you install the boot loader on a PC. When you boot, this gets executed and you get a boot prompt.

    So, your drive will only be configured one way or the other. Either GPT, or MBR. Your BIOS/EFI will then boot it (if it supports the type).

    You can use parted to create a new GPT partition table (will make your data in accessible).

    parted /dev/sdX
    mklabel gpt
    quit

    Hope that helps 🙂

20. Thanks Chris, I’ve spent a bit of time staring at the wikipedia entries on MBRs and GPTs, also did some hexdumps of the disk to see what is really going on after RHEL6 and Kubuntu installs. More later, but the MBRs are identical except for what looks like one byte before the GRUB code, and a few bytes after in the partition table. I also did a hexdump of the first 512 bytes of the EFI partition after the Kubuntu install, and it looks like there’s some sort of boot relevant stuff in there.

    Also, it seems that Kubuntu 10.10 uses Grub 2, whereas RHEL6 still uses Grub 1 and it’s not clear whether they have some of the Mac specific things that Ubuntu does.

    Learning more than I ever wanted to know about MBRs & GPTs…

    -le

21. Ok, finally made a little progress… was able to use gdisk & gptsync to mod the GPT/MBR hybrid so the Linux disk shows up at the Mac boot option screen. Still won’t boot, but at least I see the disk, so I’m thinking now I just need to reinstall the boot loading code (maybe use GRUB2?).

    The issue with the disk not showing up was a perfect storm of bugs, OS X quirks, and my not fully understanding what the tools were doing (e.g., I didn’t realize that some of the things I was doing in gdisk wiped out the boot code in the MBR). In the end, all I needed to do was use gdisk to set the type and attributes of the partitions, and then do:

    gptsync -n /dev/diskX 2+ 3

    and the disk showed up at boot time. Now if I can get the thing to actually boot…

    -le

22. So you probably need to install grub to the /boot partition (not the MBR), but you could try both.

    Fedora doesn’t use grub version 2.

    -c

23. Halle-friggin-lujah! I booted up in RHEL6 installed on the 3rd drive of my Mac Pro this morning using BIOS emulation.

    I didn’t actually have to re-install GRUB, there still were some minor mods to the MBR partition table I had to make using dd due to a buggy (or maybe non-Mac appropriate) version of gptsync.

    Bottom line is that RHEL/Fedora has a parted (or installer) bug that screws up the GPT, and you need to use the version of gptsync in rEFIt after you fix the GPT… In particular, do not use the “enhanced” version referred to in an insanely mac forum:

    http://www.mediafire.com/file/jizonwxymzr/gptsync-0.2.pkg.zip

    I want to find the minimal procedure to do this, will post in the next couple of days.

    -le

24. Great! Well done 🙂 If you post what you did here, I can put it up the top in the main post.

    -c

25. Here’s a short how-to on installing Linux (running under BIOS emulation) on a Mac Pro with multiple disks. I’ve done this with both RHEL6 and Kubuntu 10.10. RHEL6 requires a third party tool, Kubuntu does not.

    One thing to note is that fan control under Linux is less sophisticated than under OS X. Under OS X, my Mac Pro is usually almost silent, whereas with both Linux versions, the fans are always quite noticeable, although not as loud as my old Dell PC.

    My setup:

    2010 2.66 GHz 12 Core Mac Pro
    4 disks, the disks in the first two drive bays make up a RAID mirror
    Mac OS X 10.6.6 installed on the RAID mirror

    Goal: install Linux, in particular RHEL6, on the disk in the 3rd drive bay of my Mac Pro

    WARNING: mistakes or errors in the following can cause you to lose all your data. Follow the procedure at your own risk, and backup any and all disks that contain data that matters to you. This worked for me, it may not work for you.

    Preliminary Preparation for RHEL6 (you’ll need Xcode installed)
    ————————————————

    1) Download the rEFIt 0.14 source code:

    http://sourceforge.net/projects/refit/files/rEFIt/0.14/refit-src-0.14.tar.gz/download

    2) Move the downloaded tar file to some place reasonable and double-click to extract the archive contents. This should create a refit-src-0.14 directory.

    3) Open a terminal and go to the refit-src-0.14/gptsync sub-directory.

    4) Compile the source code: make -f Makefile.unix

    If you have the 10.4 SDK installed, this will fail and you’ll have to edit Makefile.unix, commenting out lines that refer to:

    /Developer/SDKs/MacOSX10.4u.sdk.

    5) This should create two executables:

    gptsync
    showpart

    6) Now in the terminal, sudo tcsh (or sudo bash) to get admin privileges, and move the executables to somewhere useful, e.g., /usr/sbin.

    Procedure for installing RHEL6
    ———————–

    1) Run the “Boot Camp Assistant” app. At the “Create or Remove a Windows Partition” dialog select the disk you will use for Linux, press “Continue” (“Erase and Create a Single Partition for Windows” should be the only possible action) and enter an administrator username/password. When you get to the “Start Windows Installation” dialog, press the “Quit & Install Later” button and exit the program.

    Using Boot Camp Assistant is probably not really necessary, everything it does should be possible with “diskutil” and “gpt” command line tools. I use it because it quickly puts the disk in a known correct state.

    2) Run the “Disk Utility” app. Select the disk you just initialized with Boot Camp Assistant and select and delete the large Windows MS-DOS (FAT) partition.

    At this point you have two options, one: 1) use the Disk Utility app (or diskutil command line tool) to create the desired partition layout for Linux, or 2) create the partition layout within the Linux installer. I assume the latter in the following.

    3) Reboot, booting off the RHEL6 installation disk (I used a DVD).

    4) Proceed with the installation and when it comes time to partition the disk, choose “Create Custom Layout”. In the free space on your intended Linux disk, create the desired partition layout. I’ve had success with a Linux+swap partition layout (i.e., “/” and swap), a boot+Linux+swap layout, and LVM.

    5) When the RHEL6 installation is finished, reboot into Mac OS X.

    6) Open a terminal and sudo tcsh or sudo bash to get admin privileges. To determine which device is associated with your Linux disk, run:

    diskutil list

    Note: the device associated with a specific disk changes across reboots in Mac OS X!

    From the commandline run:

    gptsync /dev/diskN

    where /dev/diskN is the device associated your Linux disk. If you created a Linux+swap partition layout you should see output similar to:

    Current GPT partition table:
    # Start LBA End LBA Type
    1 40 409639 EFI System (FAT)
    2 411648 3848329215 Basic Data
    3 3848329216 3907028991 Linux Swap

    Current MBR partition table:
    # A Start LBA End LBA Type
    1 1 3907029167 ee EFI Protective

    Status: MBR table must be updated.

    Proposed new MBR partition table:
    # A Start LBA End LBA Type
    1 1 409639 ee EFI Protective
    2 * 411648 3848329215 83 Linux
    3 3848329216 3907028991 82 Linux swap / Solaris

    May I update the MBR as printed above? [y/N]

    Type ‘y’ and hit return.

    That’s it. Your Linux disk should now be an option appearing as “Windows” when you hold down the option key on boot.

    NOTE: I occasionally experience a hang after selecting the RHEL6 disk and initiating the boot. There is a short amount of disk activity, and then nothing. The grey screen with the disk icons remains up. I’ve never seen this with Kubuntu, just RHEL6. RHEL6 uses an older GRUB version, so possibly a GRUB bug, although I see a similar problem mentioned in forums on dual booting Windows on Macs (e.g., http://www.insanelymac.com/forum/index.php?showtopic=201737)

    Procedure for installing Kubuntu/Ubuntu 10.10
    ————————————-

    For Kubuntu/Ubuntu installation, you don’t need the rEFIt gptsync. The following describes a Kubuntu 10.10 install, I assume Ubuntu 10.10 is essentially identical.

    1) Initialize the Linux disk as with RHEL6, but after you delete the Windows partition created by the Boot Camp Assistant app, in the Disk Utility app, create the desired Linux partition layout in the free space. I’ve only tried a Linux+swap layout, but others should work. I usually select the ExFat format for the partitions because that seems to allow more flexibility in sizing than, say, MS-DOS FAT (the format shouldn’t be critical since it will be changed below). Alternatively, use the diskutil command line tool to create the partition layout.

    2) Reboot using the Kubuntu 10.10 install disk and proceed with the installation.

    3) At the “Allocate drive space” dialog, select “Specify partitions manually (advanced)”

    4) In the “Prepare partitions” dialog, look for the Linux drive. It should contain a small ~200MB EFI partition followed by the Linux and swap partitions you created.

    5) Select the partition you want to use as the main Linux partition. A small “Edit Partition” dialog should pop up. Select “Ext4” from the “Use as:” menu, press the “Format the partition” check-box, and the mount point should be /.

    6) Select the partition you want to use as swap. In the “Edit partition” dialog, select “swap area” as the “Use as:” choice.

    7) Back at the Prepare partitions” dialog, select the “Device for boot loader installation”. You can choose the disk device, for Master Boot Record (MBR) installation or the Linux partition device for Volume Boot Record (VBR) installation (both seem to work, I usually use the MBR). For example, if your Linux disk is /dev/sdb and your Linux partition is /dev/sdb2, choose the /dev/sdb for MBR installation, /dev/sdb2 for VBR. Then press the “Install Now” button.

    8) Continue with the installation till done, and reboot.

    That’s it. Your Linux disk should now be an option appearing as “Windows” when you hold down the option key on boot.

26. Oh, I mentioned this in the Kubuntu instructions, but in my RHEL6 instructions I forgot to mention that installing the boot loader in either MBR or VBR worked.

    -le

27. Pingback: Ubuntu on Mac with multiple drives – by Larry Edwards « the blog of Chris

28. Hi Larry,

    Thanks for that, it’s great. I’ve posted it on my blog to give it more exposure and have credited you.

    https://blog.christophersmart.com/2011/03/31/linux-on-mac-with-multiple-drives-by-larry-edwards/

    -c

29. Pingback: Debian y CentOs en Mac Pro | salvaestudio

30. Has anyone considered holding the option(alt) key at boot to bring up mac’s own loader? Just a thought.