Installing Fedora Core 3 on a Supermicro SuperServer 5013C-MT with Marvell SATA with RAID

How I brought up a Supermicro SuperServer 5013C-MT on Fedora Core 3 with RAID

Hopefully someone finds this useful. These are the steps I followed; your mileage, of course, may vary. I'm writing this down because there were a couple of tricks with the Supermicro machine that I didn't see anywhere else online and I had to figure them out myself. This isn't strictly a set of installation instructions -- it's just the narrative of what I did that worked for me.

If you need help, please feel free to send me email. If it's a fairly straightforward question I will answer as soon as I can, or I can at least give you more details about my experiences.

This server has a Marvell SATA adapter. You can get the installation instructions, the drivers, and a lot of excellent information at this site:

Marvell SATA driver + Fedora Core 3/4 HOWTO

On that site, Carlos is describing the installation instructions on an HP ProLiant server. On the SuperMicro, I had to deal with a couple of additional issues.

There is no floppy drive. The instructions indicated above allude to the fact that installing with a USB disk is tricky -- it is indeed. I couldn't get that to work. I did get installation of the driver over the network to work, so I include a bit more information about that below.

Also when I was done with the install, I did a "yum update" to bring the server up to 2.6.12-1.1372. So I had to rebuild the driver for that, and while I was at it, switch over to the smp kernel. So my steps for doing that are included also.

I did the install with Linux software RAID -- I don't know of any way to make the hardware RAID work. The software RAID works fine for me -- I'm using RAID 1 because my purposes require the disks to be mirrored.

You may be able to skip the steps about compiling your own drivers by using the ones that Carlos precompiled. I compiled my own, so that's what I document here.

Prerequisites:

One Supermicro SuperServer 5013C-MT. My machine has a pair of 120G SATA drives.
Another machine running Fedora Core 3 that is accessible over the network by aforesaid Supermicro machine.
Three Fedora Core 3 install CD's.

Steps:

1. I downloaded the driver from this location to my previously existing Fedora Core 3 system and unzipped it.

[mhill@dial work]$ wget ftp://ftp.abit.com.tw/pub/download/drivers/linux/marvell/mvsata340.zip
--13:37:17-- ftp://ftp.abit.com.tw/pub/download/drivers/linux/marvell/mvsata340.zip
           => `mvsata340.zip'
Resolving ftp.abit.com.tw... 202.168.194.183
Connecting to ftp.abit.com.tw[202.168.194.183]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/download/drivers/linux/marvell ... done.
==> PORT ... done.    ==> RETR mvsata340.zip ... done.
Length: 1,200,001 (unauthoritative)

100%[==================================================>] 1,200,001     54.37K/s    ETA 00:00

13:37:47 (47.32 KB/s) - `mvsata340.zip' saved [1,200,001]

[mhill@dial work]$ unzip -d MVSATA mvsata340.zip
Archive: mvsata340.zip
   creating: MVSATA/Docs/
inflating: MVSATA/Docs/COREDriverModifications_3_2_1_to_3_4_0.pdf

... etc. (long list of files)

2. I downloaded the kernel sources that match the Fedora Core 3 install disks and installed them. You can get them from the link I used or any other convenient Fedora mirror.

[mhill@dial work]$ wget ftp://ftp.linux.ncsu.edu/pub/fedora/linux/core/3/SRPMS/kernel-2.6.9-1.667.src.rpm
--13:42:34-- ftp://ftp.linux.ncsu.edu/pub/fedora/linux/core/3/SRPMS/kernel-2.6.9-1.667.src.rpm
           => `kernel-2.6.9-1.667.src.rpm'
Resolving ftp.linux.ncsu.edu... 152.1.2.172
Connecting to ftp.linux.ncsu.edu[152.1.2.172]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/fedora/linux/core/3/SRPMS ... done.
==> PORT ... done.    ==> RETR kernel-2.6.9-1.667.src.rpm ... done.
Length: 40,291,006 (unauthoritative)

100%[==================================================>] 40,291,006    55.15K/s    ETA 00:00

13:54:46 (53.81 KB/s) - `kernel-2.6.9-1.667.src.rpm' saved [40,291,006]

3. Then I edited the /usr/src/redhat/SPECS/kernel-2.6.spec file to set buildsmp to 0, and removed %{rhbsys} from the end of the line that starts with %define release...

4. Next I compiled the kernel up to the point where the config and include files, and Module.symvers, are all completed, and pressed control-C to interrupt the build. I had to do a build with target i586 because that's the kernel that the installer users. It looked something like this:

[mhill@dial work]$ cd /usr/src/redhat/SPECS/
[mhill@dial SPECS]$ sudo rpmbuild -bc --target=i586 kernel-2.6.spec
Password:
Building target platforms: i586
Building for target i586
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.27255
+ umask 022
+ cd /usr/src/redhat/BUILD

(... lots of stuff omitted here ...)

CHK     include/asm-i386/asm_offsets.h
UPD     include/asm-i386/asm_offsets.h
CHK     include/linux/compile.h
UPD     include/linux/compile.h
arch/i386/kernel/signal.c: In function `setup_sigcontext':
arch/i386/kernel/signal.c:269: warning: unused variable `eflags'

(then I pressed control-C)

make[1]: *** [arch/i386/kernel/ptrace.o] Interrupt
make: *** [arch/i386/kernel] Interrupt

5. Then I made the link to linux-2.6 in the src directory:

[mhill@dial SPECS]$ cd /usr/src/
[mhill@dial src]$ ln -s redhat/BUILD/kernel-2.6.9/linux-2.6.9 linux-2.6

6. At this point I was ready to build the driver.

[mhill@dial LinuxIAL]$ sudo bash build.sh
Password:
Generate driver for Marvell PCI-X to Serial ATA adapters - kernel version 2.6.
rm -f *.o *.ko .*.cmd .*.o.flags *.mod.c
Compiling DebugFull...
Building modules, stage 2.
rm -f *.o *.ko .*.cmd .*.o.flags *.mod.c
Compiling DebugError...
Building modules, stage 2.
rm -f *.o *.ko .*.cmd .*.o.flags *.mod.c
Compiling Free...
Building modules, stage 2.
[mhill@dial LinuxIAL]$ ls -l mv_sata.ko
-rw-r--r-- 1 root root 1057583 Aug 27 14:34 mv_sata.ko
[mhill@dial LinuxIAL]$ strings mv_sata.ko |grep magic
vermagic=2.6.9-1.667 586 REGPARM 4KSTACKS gcc-3.4

The strings command verifies that I built what I thought I was going to -- a driver for 2.6.9-1.667 for the i586.

7. So at this point, I have a driver that's appropriate to work with the Fedora Core 3 installer.

Next I connected the ethernet to the machine.

I inserted the Fedora Core 3 installer CD #1, and booted the machine up. At the install prompt, I typed

linux askmethod

I took the defaults (pressing Enter) up until this screen displayed, prompting me for which type of media to use:

8. Now this was a bit of a cheat. From that menu, I selected FTP.

I selected eth0, and then selected dynamic IP configuration with DHCP. I selected OK (make sure you select OK, so that the interface gets configured with DHCP.)

Then the next screen requested information about the FTP server. At that point, I just selected Back, Back, Back until I was back at the media selection screen again.

Now I have the network interface configured. So I went ahead and selected Local CDROM from the media menu, skipped the media check (since I already knew my CD's were ok) and continued up until I saw this screen ("No hard drives have been found"):

9. At this point I pressed control-alt-F2 to get a shell prompt, and executed these commands to ftp the driver over to my /tmp directory:

-/bin/sh-3.00# sftp mhill@dial... (the full domain name of the machine where I built the driver)
(answered the password prompt)
sftp> cd work/MVSATA/LinuxIAL
sftp> lcd /tmp
sftp> get mv_sata.ko
(download messages)
sftp> quit

And then I loaded the driver up:

-/bin/sh-3.00# modprobe /tmp/mv_sata.ko

and ran dmesg just to check that the end of the log says that it worked.

-/bin/sh-3.00# dmesg

I got a message saying that scsi disk sda and sdb were attached, and lots of messages about the Marvell SCSI to SATA adapter, so things looked good. If you get an error message complaining about the wrong version of the driver, or something like that, you need to check to make sure you built a driver that matched your kernel version, etc.

10. So I was all set to go with the driver loaded. So I pressed control-alt-F1 to get back to the installer, and answered "no" to the question about selecting drivers (I didn't have to anymore, since I just loaded them manually), and let the installer come up into the familiar graphical install screen:

I proceeded to the screen that asks about Disk Partitioning, and chose "Manually partition with Disk Druid". When I got the error saying "the partition table on device sda/sdb was unreadable" I just told it to go ahead and erase all data. This is a brand new machine.

I partitioned sda with 500 meg space for /boot, and 1000 meg swap. On sdb I partitioned 500 meg for /spare, and 1000 meg swap (just so that the disks would look identical). That got me up to a partition table looking like this:

11. For each drive, I pressed the RAID button and set up a "software RAID" partition with "fill to maximum allowable size", to use all the remaining space on the disk. Which resulted in this:

12. And then I clicked on RAID again, and selected "Create a RAID device", with a mount point of /, and RAID1, and then I ended up with this. Thus endeth the partitioning.

13. At this point it proceeded like a fairly normal Fedora install -- I answered all the questions my usual ways, put in all the requested CD's, and when I was asked to reboot -- STOP! DON'T REBOOT!!! See, I told you not to reboot. (Actually the first time I did this as a dry run, I did reboot, and man, what a pain. So just don't reboot.)

Don't do what this screen says:

14. Now I needed to install an i686 version of the driver on this machine, since that was the kernel that was going to try to boot up when I rebooted. So I went back to my trusty build machine (dial), and did this:

[mhill@dial LinuxIAL]$ cd /usr/src/redhat/SPECS/
[mhill@dial SPECS]$ sudo rpmbuild -bc --target=i686 kernel-2.6.spec
Password:
Building target platforms: i686
Building for target i686
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.43666
+ umask 022
+ cd /usr/src/redhat/BUILD

(... omit lots of messages ...)

CHK include/linux/compile.h
UPD include/linux/compile.h
arch/i386/kernel/signal.c: In function `setup_sigcontext':
arch/i386/kernel/signal.c:269: warning: unused variable `eflags'

(... then I pressed control-C to interrupt)

make[1]: *** wait: No child processes. Stop.
make[1]: *** Waiting for unfinished jobs....
make[1]: *** wait: No child processes. Stop.
make: *** [arch/i386/kernel] Error 2

[mhill@dial SPECS]$ cd ~/work/MVSATA/LinuxIAL/
[mhill@dial LinuxIAL]$ sudo bash build.sh
Password:
Generate driver for Marvell PCI-X to Serial ATA adapters - kernel version 2.6.
rm -f *.o *.ko .*.cmd .*.o.flags *.mod.c
Compiling DebugFull...
Building modules, stage 2.
rm -f *.o *.ko .*.cmd .*.o.flags *.mod.c
Compiling DebugError...
Building modules, stage 2.
rm -f *.o *.ko .*.cmd .*.o.flags *.mod.c
Compiling Free...
Building modules, stage 2.
[mhill@dial LinuxIAL]$ ls -l mv_sata.ko
-rw-r--r-- 1 root root 1048783 Aug 27 15:48 mv_sata.ko
[mhill@dial LinuxIAL]$ strings mv_sata.ko |grep magic
vermagic=2.6.9-1.667 686 REGPARM 4KSTACKS gcc-3.4

Now you see that it's an i686 build of the driver.

15. I went back to the server machine being installed and pressed control-alt-F2 to get a shell prompt, and sftp'd the driver to /tmp the same way I did last time in step 9 but instead of doing the modprobe, I just copied the driver over to the disk like so:

-/bin/sh-3.00# cp /mnt/fd/mv_sata.ko /mnt/sysimage/lib/modules/2.6.9-1.667/kernel/drivers/scsi/

16. Then I ran the steps exactly as provided at Carlos' site to modify my modprobe.conf file, run depmod, and run mkinitrd.

chroot /mnt/sysimage
echo "alias scsi_hostadapter mv_sata" >> /etc/modprobe.conf
/sbin/depmod -ae -F /boot/System.map-2.6.9-1.667 2.6.9-1.667
/sbin/mkinitrd -v -f /boot/initrd-2.6.9-1.667.img 2.6.9-1.667

17. Only then did I press control-alt-F7 to return to the installer, remove the CD from the CD drive, and click on the reboot button. When the blue screen appeared after rebooting to select a kernel, I pressed the space bar immediately, and selected the "up" kernel from the menu (remember we haven't built a driver for the smp kernel yet... that's still coming).

18. The machine rebooted and let me finish the last couple of install steps (when the kudzu prompt came up, I pressed the space bar and selected "configure", which gave me a reassuring message that it knew about the Marvell driver.) Eventually I got a login prompt and logged in as myself.

19. I added myself to the list of sudoers by logging in as root on another virtual console, and editing /etc/sudoers appropriately.

20. I added the GPG keys for the yum repository in preparation for upgrading to the latest Fedora patches, by doing

sudo rpm --import /usr/share/rhn/RPM-GPG-KEY-fedora

This is a good place to visit if you want more information about how to set up yum.

21. Then I ran a full yum update, to get the latest patches from Fedora. This is another one of those places where, when you're done, you should not reboot yet. There is more to come.

sudo yum update

22.   At this point, all the files have been installed for a new kernel (in this case, 2.6.12-1.1372_FC3), but the system did not yet have the corresponding version of mv_sata.ko installed. I downloaded and installed the sources for this kernel from one of the mirrors:

[mhill@localhost work]$ wget ftp://ftp.linux.ncsu.edu/pub/fedora/linux/core/updates/3/SRPMS/kernel-2.6.12-1.1372_FC3.src.rpm
--22:28:24-- ftp://ftp.linux.ncsu.edu/pub/fedora/linux/core/updates/3/SRPMS/kernel-2.6.12-1.1372_FC3.src.rpm
           => `kernel-2.6.12-1.1372_FC3.src.rpm'
Resolving ftp.linux.ncsu.edu... 152.1.2.172
Connecting to ftp.linux.ncsu.edu[152.1.2.172]:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done. ==> CWD /pub/fedora/linux/core/updates/3/SRPMS ... done.
==> PORT ... done.    ==> RETR kernel-2.6.12-1.1372_FC3.src.rpm ... done.
Length: 38,404,429 (unauthoritative)

100%[====================================>] 38,404,429   157.97K/s    ETA 00:00

22:32:30 (153.25 KB/s) - `kernel-2.6.12-1.1372_FC3.src.rpm' saved [38,404,429]

[mhill@localhost work]$ sudo rpm -ivh kernel-2.6.12-1.1372_FC3.src.rpm
Password:
   1:kernel                 ########################################### [100%]

23. I downloaded the version of the driver that was patched for FC4 from this location. This was necessary because the change in this patch applies to the 2.6.12-1.1372 kernel that is in the latest updates for FC3.

[mhill@localhost work]$ wget http://www.keffective.com/mvsata/FC3/mvsata-3.4.2-1.0_fc4.tgz
--22:56:19-- http://www.keffective.com/mvsata/FC3/mvsata-3.4.2-1.0_fc4.tgz
           => `mvsata-3.4.2-1.0_fc4.tgz'
Resolving www.keffective.com... 68.84.232.131
Connecting to www.keffective.com[68.84.232.131]:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 710,302 [application/x-gzip]

100%[====================================>] 710,302       30.83K/s    ETA 00:00

22:56:37 (38.97 KB/s) - `mvsata-3.4.2-1.0_fc4.tgz' saved [710,302/710,302]

24. I did an rpmbuild of the new kernel sources in order to generate the appropriate configuration files.

[mhill@localhost work]$ cd /usr/src/redhat/SPECS/
[mhill@localhost SPECS]$ ls
kernel-2.6.spec
[mhill@localhost SPECS]$ sudo rpmbuild --bc --target=i686 kernel-2.6.spec
Password:
Building target platforms: i686
Building for target i686
Executing(%prep): /bin/sh -e /var/tmp/rpm-tmp.82441
+ umask 022
+ cd /usr/src/redhat/BUILD

(... this one I had to let run all the way through, because the Makefiles changed so that Module.symvers isn't written out until the end ...)

25. I created a new link to the 2.6.12 sources in /usr/src:

[mhill@localhost SPECS]$ cd /usr/src
[mhill@localhost src]$ sudo ln -s redhat/BUILD/kernel-2.6.12/linux-2.6.12 linux-2.6

26. Then I untarred the MVSATA driver source and built it with the new configuration.

[mhill@localhost SPECS]$ cd ~/work
[mhill@localhost work]$ ls
kernel-2.6.12-1.1372_FC3.src.rpm mvsata-3.4.2-1.0_fc4.tgz
[mhill@localhost work]$ tar xvzf mvsata-3.4.2-1.0_fc4.tgz
mvsata-3.4.2-1.0_fc4/
mvsata-3.4.2-1.0_fc4/CoreDriver/

(etc.)

[mhill@localhost work]$ cd mvsata-3.4.2-1.0_fc4/LinuxIAL
[mhill@localhost LinuxIAL]$ sudo bash build.sh
Generate driver for Marvell PCI-X to Serial ATA adapters - kernel version 2.6.9-1.667.
Compiling DebugFull...
make -C /usr/src/linux-2.6 M=/home/mhill/work/mvsata-3.4.2-1.0_fc4/LinuxIAL

(... lots of build messages omitted... )

Building modules, stage 2.
MODPOST
CC      /home/mhill/work/mvsata-3.4.2-1.0_fc4/LinuxIAL/mv_sata.mod.o
LD [M] /home/mhill/work/mvsata-3.4.2-1.0_fc4/LinuxIAL/mv_sata.ko
make[1]: Leaving directory `/usr/src/redhat/BUILD/kernel-2.6.12/linux-2.6.12'
mv_sata.ko copied to ./../build/Linux/Free
[mhill@localhost LinuxIAL]$ cd ~/work/mvsata-3.4.2-1.0_fc4/build/Linux/Free/
[mhill@localhost Free]$ ls
mv_sata.ko
[mhill@localhost Free]$ strings mv_sata.ko |grep magic
vermagic=2.6.12-1.1372_FC3.rootsmp SMP 686 REGPARM 4KSTACKS gcc-3.4

27. Then I copied that driver to the appropriate location for the new smp kernel.

[mhill@localhost LinuxIAL]$ sudo cp ../build/Linux/Free/mv_sata.ko /lib/modules/2.6.12-1.1372_FC3smp/kernel/drivers/scsi

28. Then I reran depmod and mkinitrd again...

[mhill@localhost LinuxIAL]$ sudo /sbin/depmod -ae -F /boot/System.map-2.6.12-1.1372_FC3smp 2.6.12-1.1372_FC3smp
[mhill@localhost LinuxIAL]$ sudo /sbin/mkinitrd -v -f /boot/initrd-2.6.12-1.1372_FC3smp.img 2.6.12-1.1372_FC3smp
Creating initramfs
Looking for deps of module scsi_mod

(... omitted lots of mkinitrd messages here ...)

29. Then I was done! Rebooted with the new kernel.

Contact:

Mark Hill (mark.hill.webcontact@gmail.com)

Other credits:

I purchased this machine from Pixel USA in San Jose. They supply a lot of the equipment we use here at Epicentre and they treat us well.

This work was done on behalf of MaxBidPrice, which is designing an interesting online auction optimizer that you should check out.