# fedora on zfs

## hp proliant

it is my experience that fedora server, up to version 42, does not play nicely with hp proliant servers which have more than 8 hard disk drives and hp standard raid controllers. i have had only positive experiences when there are 8 or fewer drives and hp hardware raid is used in any of the supported raid configurations (0, 1, 5, 6, 50, 60). however on systems with 12 × 3.5" or 25 × 2.25" drives, when any flavour of hardware raid is enabled it will inevitably result in a system whose boot or root partition is lost during a reboot and the system requires a full os reinstall before it will ever boot successfully again. this experience has been repeated on numerous occassions with multiple hp proliant servers gen 7 through 10 inclusive, using both internally mounted sas drives and when an external sas enclosure is connected.

the instructions below document successful installs of fedora server 41 on a root zfs filesystem spanning all drives on proliant many-drive systems.

the only difference between these instructions and those in the [fedora-on-zfs readme](https://github.com/gregory-lee-bartholomew/fedora-on-zfs/blob/main/README.md) is that they include the bios raid configuration, use `raidz3` (rather than `mirror`) raid configuration and explicitly set the list of target installation drives to those found on typical hp proliant servers containing more than 8 drives.

### gen 10

#### dl380

##### p816i-a (raid controller)

put the raid controller into *enable software raid* mode. this is hpe terminology for *bypass the hardware raid controller and give the os direct access to the spinning rust*. it is as close as this controller gets to initiator-target ^(it) mode

references:
- https://support.hpe.com/hpesc/public/docDisplay?docId=a00026445en_us&docLocale=en_US
- https://github.com/gregory-lee-bartholomew/fedora-on-zfs/issues/7
- https://community.hpe.com/t5/proliant-servers-ml-dl-sl/booting-from-a-hba-drive/td-p/7197337

> 1. Reboot the server.
> 2. Start UEFI System Utilities by pressing F9 (System Utilities) during POST.
> 3. Select System Configuration > BIOS/Platform Configuration (RBSU) > Storage Options > SATAController Options > Embedded SATA Configuration > Smart Array SW RAID Support and pressthe Enter key.
> 4. In the SATA Controller Options screen, for the Embedded SATA configuration option, select SmartArray SW RAID Support from the drop-down menu and click OK. If user see the warning "Important: Smart Array SW RAID is not supported when the Boot Mode isconfigured in Legacy BIOS Mode", click OK.
> 5. Press F12: Save and Exit.
> 6. Save changes by clicking Yes-Save Changes.
> 7. Click Reboot.

##### install fedora on zfs

> [!warning]
> we're selecting all disks returned by `ls /dev/sd{a..l}` when running fedora workstation live. modify accordingly if `/dev/sda`, `/dev/sdb`, ..., `/dev/sdk`, `/dev/sdl` is not the intended target disk list.

- boot from latest fedora workstation usb (tested with *fedora media writer* and *fedora workstation 42*).
> [!tip]
> workstation rather than server is required in order to get easy access to the fedora on zfs prerequisites
- fedora on zfs

  # install fedora on zfs
  sudo -i
  git clone \
      https://github.com/gregory-lee-bartholomew/fedora-on-zfs.git

  # install prerequisites
  ./fedora-on-zfs/prereqs

  # modify config for a many-drive system
  sed \
      -i \
      -e "s/'mirror'/'raidz3'/" \
      ./fedora-on-zfs/config/evariables.conf

  # select target disks (/dev/sda to /dev/sdl inclusive)
  disks=( $(ls /dev/sd{a..l}) )

  # run fedora on zfs
  cd ./fedora-on-zfs
  ./install fedora-disk-server.ks ${disks[@]}

> [!note]
> when prompted:
> > UEFI detected, update this system's default boot order? [y/n]:
>
> choose <kbd>y</kbd>.
- disable secure boot (or update the trusted secure boot keys)
> [!tip]
> if the system fails to boot following the os install and reboot, select <kbd>f11</kbd> *choose boot device* and select one of the entries created by *fedora on zfs*. if this immediately returns you to the boot device selection screen, it's an indication that secure boot is enabled but the fedora secure boot keys are not included in the bios secure boot key database. either disable secure boot in the bios, or update it's trusted key database.

### gen 8

#### dl380e

##### p420 (raid controller)

- enable ahci in the bios
<kbd>f9</kbd> > system options > sata controller options > embedded sata configuration > enable sata ahci support

- switch on hba (whilst booted into the fedora workstation livecd)

  # install smart storage administrator cli
  sudo dnf install https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p1857046646/v183344/ssacli-4.21-7.0.x86_64.rpm
  
  # enable hba mode
  sudo ssacli controller slot=2 modify hbamode=on
  
  # 
  sudo ssacli controller slot=2 show
  sudo ssacli controller slot=2 physicaldrive all show
  
  sudo ssacli controller slot=2 physicaldrive 1I:1:1 modify bootvolume=primary


  sudo dnf install https://downloads.hpe.com/pub/softlib2/software1/pubsw-linux/p803169573/v193307/hp-health-10.93-307.4.rhel8.x86_64.rpm

- fedora on zfs

  # install fedora on zfs
  sudo -i
  git clone \
      https://github.com/gregory-lee-bartholomew/fedora-on-zfs.git

  # install prerequisites
  ./fedora-on-zfs/prereqs

  # modify config to enable bios boot (no uefi on gen 8)
  sed \
      -i \
      -e "s/# BIOSBOOT/BIOSBOOT/" \
      ./fedora-on-zfs/config/evariables.conf

  # install fedora on zfs to the first two disks (zfs mirror)
  cd ./fedora-on-zfs
  ./install fedora-disk-server.ks /dev/sdc /dev/sdd

- update the p420i firmware

  sudo dnf install https://downloads.hpe.com/pub/softlib2/software1/sc-linux-fw-array/p2102220776/v140762/rpm/RPMS/x86_64/hp-firmware-smartarray-46a4d957a7-8.32-1.1.x86_64.rpm
  rpm -ql hp-firmware-smartarray-46a4d957a7
  sudo /usr/lib/x86_64-linux-gnu/hp-scexe-compat/CP033363.scexe


- clone the fzfs boot partition to the sd card

  sudo dd if=/dev/sda of=/dev/sdz bs=1M status=progress

*[hpe]: hewlett packard enterprise