Scheduled Maintenance: Over the course of a few days we will be addressing issues with the search backend. General search, newposts, yourposts, and other search driven queries may fail during the update. Details and discussion here: viewtopic.php?t=159736

 

 

 

Disk order and preseed recipe

Ask for help with issues regarding the Installations of the Debian O/S.
Post Reply
Message
Author
aswen
Posts: 3
Joined: 2024-07-05 07:47
Has thanked: 1 time

Disk order and preseed recipe

#1 Post by aswen »

Dear fellow Debian users,

[tl;dr] Please advice me on how to automatically figure out which disk (actually RAID array) is which during the early_command stage of an automated install while the RAID controller presents the disks to the kernel in random order and I want to use specific arrays for specific logical volumes.
At this moment, my question is: How can I make the arrays visible in /dev/block during the early_command stage?
Or, is there any other way to get the correct partitions on the desired arrays?

Background
I have to install many servers. Any manual action in the installation process should be automated. We have different types of servers, from different manufacturers. Some servers have different types of disks in them and we use the RAID controllers to create arrays (Dell, for example, calls them "Virtual Disks", other manufacturers refer to them as "Logical disks", etc. I call them arrays). Our relational database machines are designed to have three arrays:
  • OS+binlogs (SSD storage ~900G)
  • DB data (SSD storage ~3,5T)
  • Backups (HDD storage ~4T)
Before you ask: the two SSD storages have different kinds of SSD's under them. One is more suitable for write intensive workloads and one more for read intensive workloads.
We use Redfish to create those RAID arrays. I will leave out the details of how we do that, but that works.
The RAID arrays are created in the order as listed above.

The problem
Even though we create the arrays in the order we like them, they're not configured in the desired order. And that order is not predictable. So, if we use /dev/sda, /dev/sdb and /dev/sdc, we may end up with our OS+binlogs volume on the spinning backup disks, while our DB data volume is written on the 900G SSD's and the backups on the SSD's intended for data. (or any variant you can think of). In the images below you'll another example of what we don't want:

ImageImage

What I tried to circumvent this:
  • The RAID controller assigns ID's to the arrays. For a moment we thought these affected how the controller presented the arrays to the driver. The ID's were assigned in decreasing order, so I created the arrays in opposite order. No luck, obviously.
  • Dell seems to support a Firmware Device Order option (which is about the only thing you cannot enable using RedFish!) which promises to address exactly this problem. We tried and proved it's not working.
  • I tried to add some code to the early_command script we have. That code could use the `/dev/block/sd?/queue/rotational` and `/dev/block/sd?/size` information to determine which disk is for what. I reasoned: the smallest SSD is always for the OS, the biggest for DB data and the rotating one for backups. Sadly, despite writing a beautiful script (see below), the megaraid_sas driver isn't loaded yet so `/dev/block` is still empty at this stage. I then tried to download a megaraid_sas.ko file to load but that fails as well. (I made sure it's the same kernel as busybox is running on then, but this is were I have not enough knowledge of the process to fix the load issue).
  • I asked a similar question to this one in the Dell community forum where I focus on presenting the arrays to the OS in the right order. This question is about my alternative approach: How can I fix this manufacturer-independent in the Debian Installer?
Used hardware
We use Dell Poweredge r660xs and Fujitsu Primergy rx2530M7s servers. Both are shipped with Broadcom RAID controllers which work nearly identical.

Details about loading the megaraid_sas driver:
I added this to the early_command script:

Code: Select all

wget -q $recipe_url_base/megaraid_sas.ko
insmod megaraid_sas.ko
Gives

Code: Select all

insmod: ERROR: could not insert module megaraid_sas.ko: Invalid module format.
I checked and busybox runs:

Code: Select all

Linux hostname 6.1.0-22-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.94-1
and the driver I tried came from an installed Debian server:

Code: Select all

/lib/modules/6.1.0-22-amd64/kernel/drivers/scsi/megaraid/megaraid_sas.ko
I made sure that scsi_mod was loaded already. (megaraid_sas depends on that)
Attachments
345375557-d04a5565-30a6-499d-a511-0237e353560a.png
345376117-f57fcd8d-1d4b-4302-a83f-dbfd0e794d41.png
345376117-f57fcd8d-1d4b-4302-a83f-dbfd0e794d41.png (15.25 KiB) Viewed 211 times
Last edited by aswen on 2024-07-10 11:43, edited 1 time in total.

aswen
Posts: 3
Joined: 2024-07-05 07:47
Has thanked: 1 time

Re: Disk order and preseed recipe

#2 Post by aswen »

Hi,
I should have better RTFM and found my problem myself: I should run the script to find the arrays in the `partman/early_command` stage, not in the `preseed/early_command` stage, which I learned today :)

For the sake of documentation for others, here my example:
Replace FQDN and URL with your webserver and URL

Code: Select all

d-i partman/early_command string \
  wget -O /root/find_raid_arrays.sh URL ;\
  chmod +x /root/find_raid_arrays.sh ;\
  /root/find_raid_arrays.sh >> /root/find_raid_arrays.log 2>&1

...
# USE THE FOLLOWING ONLY IF YOU ARE VERY SURE!!!!
d-i partman-lvm/device_remove_lvm boolean true
d-i partman-auto/purge_lvm_from_device boolean true
# The same applies to pre-existing software RAID array:
d-i partman-md/device_remove_md boolean true
# It's ok to have /boot in the LVM
d-i partman-auto-lvm/no_boot boolean true

d-i partman-basicfilesystems/no_mount_point boolean true
d-i partman/mount_style select label

# This makes partman automatically partition without confirmation, provided
# that you told it what to do using one of the methods above.
d-i partman-lvm/confirm_nooverwrite boolean true
d-i partman-lvm/confirm boolean true
d-i partman-partitioning/confirm_write_new_label boolean true
d-i partman/choose_partition select Finish
d-i partman/confirm_nooverwrite boolean true
d-i partman/confirm boolean true
The script:

Code: Select all

#!/bin/sh
# SETTINGS
me=$(basename $0)

# FUNCTIONS

log () { printf '%s %s\n' "$me: " "$@"; }

# Function to feed a variable to the debian-installer
# so it can be used later.
db_really_set() {
  var=$1 ; shift
  val=$1 ; shift
  seen=$1 ; shift

  db_register debian-installer/dummy "$var"
  db_set "$var" "$val"
  db_subst "$var" ID "$var"
  db_fset "$var" seen "$seen"
}

get_disk_details() {
  local disk=$1
  local rotational=$(cat /sys/block/$disk/queue/rotational)
  local size=$(cat /sys/block/$disk/size)
  echo "$rotational $size"
}

# SCRIPT
log "Started ${me}."

# This is a debconf-compatible script
. /usr/share/debconf/confmodule

# Set the base url for recipe downloads.
recipe_url_base="http://FQDN/recipes"

# Store this in database for use in the postinstall script.
recipe_file='/root/partition_recipe'

# Get the device_role from the preseed file
device_role=$(debconf-get our_own_settings/device_role)
partitionlayout=$device_role

if [ "$device_role" != "virtual-machine" ];then
  log "Begin partition creating the partitionrecipe"
  echo -n 'default :: 1 1 1 free $iflabel{ gpt } method{ biosgrub } . ' > $recipe_file

  if [ -d "/sys/firmware/efi" ] ;then
    log "This is running in UEFI mode, adding an EFI partition."
    echo -n '256 256 256 free $reusemethod{ } method{ efi } format{ } . ' >> $recipe_file
  fi
else
    i_am_virtual=1
fi

# Iterate over all sdX devices
for disk in /sys/block/sd*; do
  disk=$(basename $disk)
  details=$(get_disk_details $disk)
  log "DETAILS: $details"
  rotational=$(echo $details | cut -d ' ' -f 1)
  log "ROTATIONAL: $rotational"
  size=$(echo $details | cut -d ' ' -f 2)
  log "SIZE: $size"

  if [ "$rotational" -eq 0 ]; then
    log "$disk is a SSD"
    printf '%d %s\n' $size "/dev/$disk" >> /root/unsorted_ssds
  else
    log "$disk is a HDD"
    printf '%d %s\n' $size "/dev/$disk" >> /root/unsorted_hdds
  fi
  echo
done

OS_ARRAY=$(sort -n /root/unsorted_ssds|head -n 1| cut -d ' ' -f 2)
DATA_ARRAY=$(sort -n /root/unsorted_ssds| tail -n 1 | cut -d ' ' -f 2)
BACKUPS_ARRAY=$(sort -n /root/unsorted_hdds| head -n 1 | cut -d ' ' -f 2)

log "OS_ARRAY=$OS_ARRAY"
log "DATA_ARRAY=$DATA_ARRAY"
log "BACKUPS_ARRAY=$BACKUPS_ARRAY"

log "Downloading the ${partitionlayout} part of the recipe."
wget -q $recipe_url_base/$partitionlayout -O - |\
  sed \
    -e "s?OS_ARRAY?$OS_ARRAY?g" \
    -e "s?DATA_ARRAY?$DATA_ARRAY?g" \
    -e "s?BACKUPS_ARRAY?$BACKUPS_ARRAY?g" |\
  tr '\n' ' ' >> $recipe_file

log "The installer will use $recipe_file as Disk recipe."
db_really_set partman-auto/expert_recipe_file $recipe_file yes

log "Finished"

User avatar
sunrat
Administrator
Administrator
Posts: 6950
Joined: 2006-08-29 09:12
Location: Melbourne, Australia
Has thanked: 122 times
Been thanked: 570 times

Re: Disk order and preseed recipe

#3 Post by sunrat »

Please use code tags for terminal text, logs etc. Fixed it for you this time.
“ computer users can be divided into 2 categories:
Those who have lost data
...and those who have not lost data YET ”
Remember to BACKUP!

aswen
Posts: 3
Joined: 2024-07-05 07:47
Has thanked: 1 time

Re: Disk order and preseed recipe

#4 Post by aswen »

sunrat wrote: 2024-07-08 22:28 Please use code tags for terminal text, logs etc. Fixed it for you this time.
Ha, thanks for pointing that out. I searched for (and obviously overlooked) that... I updated my 1st post as well, couldn't find a way to do "inline" code (like we can do in MD: "the IP address was `10.10.10.10`"). But this looks way better already.

Aki
Global Moderator
Global Moderator
Posts: 3352
Joined: 2014-07-20 18:12
Location: Europe
Has thanked: 92 times
Been thanked: 442 times

Re: Disk order and preseed recipe

#5 Post by Aki »

Hello,
aswen wrote: 2024-07-08 15:38 I should have better RTFM and found my problem myself: I should run the script to find the arrays in the `partman/early_command` stage, not in the `preseed/early_command` stage, which I learned today :)
[..]
Thanks for sharing the solution.

Please mark the discussion as "solved" adding the tag "[Solved]" to the subject of the first post.
⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Debian - The universal operating system
⢿⡄⠘⠷⠚⠋⠀ https://www.debian.org
⠈⠳⣄⠀

Post Reply