Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

HOWTO: Build an upstream kernel package (foolproof script)

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
Kringel
Posts: 44
Joined: 2012-10-20 10:33

HOWTO: Build an upstream kernel package (foolproof script)

#1 Post by Kringel »

The following shellscript will build a debian kernel package from upstream sources:

Code: Select all

#! /usr/bin/env bash
#
# All output is stored in the current working directory. It is strongly
# recommended to run it in an empty directory.
#
# This script by default does the following:
#  - download the upstream sources of the linux kernel
#  - verify they are signed with a trusted GPG-key
#  - use the configuration of your currently running kernel
#  - build a Debian-kernel package with
#
# Each step is called as function at the end of the script. You can
# easily comment them out to skip them or to run manual steps (for
# example changing the kernel configuration with "make menuconfig").
#
#
# Packages required to build the kernel package:
#      kernel-package fakeroot build-essential devscripts
#
# If you want to change the configuration manually you need
#  for make gconfig: libgtk2.0-dev libglade2-dev
#  for make menuconfig: libncurses5-dev

set -e

# -------------------- setup --------------------

# Fingerprint of a trusted key the kernel is signed with.
# See http://www.kernel.org/signature.html
#     http://lwn.net/Articles/461647/
# ATTENTION: Make sure you really trust it!
TRUSTED_FINGERPRINT='ABAF 11C6 5A29 70B1 30AB  E3C4 79BE 3E43 0041 1886'

# The upstream kernel version to downloaded and build.
KERNEL_VERSION=3.8

# A string that is appended to the version. For restrictions see the --append-to-version option of make-kpg.
VERSION_POSTFIX=-upstream

# URL where the archive and sources are located.
SOURCE_URL_BASE=http://www.kernel.org/pub/linux/kernel/v3.x

# Server used to get the trusted key from.
KEYSERVER=hkp://pool.sks-keyservers.net

# Set to yes if you want to build only the modules that are currently
# loaded Speeds up the build. But modules that are not currently
# loaded will be missing!  Only usefull if you really have to speed up
# the build time and the kernel is intended for the running system and
# the hardware is not expected to change.
BUILD_ONLY_LOADED_MODULES=no

# -----------------------------------------------

# Remove spaces from the fingerprint to get a "long key ID" (see gpg manpage)
TRUSTEDLONGID=`echo $TRUSTED_FINGERPRINT |  sed "s/ //g"`

# Directory that is used by this script to store the trusted GPG-Key (not your personal GPG directory!)
export GNUPGHOME=./kernelkey

# Downloads the trsuted key from a keyserver. Uses the trusted fingerprint to find the key.
function RecvKey()
{
   echo "Recieving key $TRUSTED_FINGERPRINT from the keyserver..."
   [ ! -d $GNUPGHOME ] || rm -rf $GNUPGHOME # makes sure no stale keys are hanging around
   mkdir $GNUPGHOME
   chmod og-rwx $GNUPGHOME
   gpg --keyserver $KEYSERVER --recv-keys $TRUSTEDLONGID
}

# Downloads the sources and their signature file.
function DownloadSources()
{
   wget $SOURCE_URL_BASE/linux-$KERNEL_VERSION.tar.xz
   wget $SOURCE_URL_BASE/linux-$KERNEL_VERSION.tar.sign
}

# Verifies the downloaded sources are signed with the trusted key and extracts them.
function VerifyExtract()
{
   echo "Extracting downloaded sources to tar..."
   [ -f linux-$KERNEL_VERSION.tar ] || unxz --keep linux-$KERNEL_VERSION.tar.xz
   echo "Verifying tar is signed with the trusted key..."
   gpg -v --trusted-key 0x${TRUSTEDLONGID:24} --verify linux-$KERNEL_VERSION.tar.sign

   [ ! -d linux-$KERNEL_VERSION ] || rm -rf linux-$KERNEL_VERSION

   echo "Extracting tar..."
   tar -xf linux-$KERNEL_VERSION.tar
   rm linux-$KERNEL_VERSION.tar
}

# Copies the configuration of the running kernel and applies defaults to all settings that are new in the upstream version.
function SetCurrentConfig()
{
   pushd ./linux-$KERNEL_VERSION

   # Copy settings of the currently running kernel
   cp /boot/config-$(uname -r) ./.config
   
   # Use the copied configuration and apply defaults to all new settings
   yes "" | make oldconfig

   if [ yes == $BUILD_ONLY_LOADED_MODULES ]
   then
      echo "Disabling modules that are not loaded by the running system..."
      make localmodconfig
   fi

   popd
}

function Build()
{
   pushd ./linux-$KERNEL_VERSION

   # See the following links for more information:
   # http://www.debian.org/doc/manuals/debian-faq/ch-kernel.en.html
   # http://www.debian.org/releases/stable/amd64/ch08s06.html.en
   time fakeroot make-kpkg --jobs `getconf _NPROCESSORS_ONLN` --append-to-version "$VERSION_POSTFIX" --initrd kernel_image
   popd

   echo "Congratulations! You just build a linux kernel."
   echo "Use the following command to install it: dpkg -i linux-image-${KERNEL_VERSION}${VERSION_POSTFIX}*.deb"
}

RecvKey
DownloadSources
VerifyExtract
SetCurrentConfig
Build
Rationale:
After installing the wheezy beta my system froze multiple days in a row multiple times per day (but that is another thread). So I decided to try an upstream kernel. It took me a while to search for documentation and to find a good way to build the kernel package. The result is the script you see above. I post it here to spare you from repeating all the steps I (and a few thousand other debian users) already did. I am a big fan of automating things that are needed repeatedly :)

EDIT: Fixed some minor problems.
EDIT: Added an option to build only loaded modules.
EDIT Feb 24 2013: Updated for kernel Version 3.8, GPG key updated, got rid of GPG warning, corrected some comments
Last edited by Kringel on 2013-02-24 10:24, edited 4 times in total.

vbrummond
Posts: 4432
Joined: 2010-03-02 01:42

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#2 Post by vbrummond »

Fun stuff. I would test it if I didn't already have my own methods of doing so. :D Looks solid to me though, thanks for contributing.
Always on Debian Testing

Ibidem
Posts: 160
Joined: 2010-12-24 18:28

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#3 Post by Ibidem »

Any reason for not going "make deb-pkg" (the upstream way) ?

Also, if you don't have any extra removable peripherals (ie, all the modules you'll use are loaded), make localmodconfig will really speed up the build (I think better than halve build time...).
But all in all, looks good.
Thinkpad X100e/Debian Squeeze (All reposiories enabled)/Linux 3.4.11:
1GB RAM/1.6GHz Neo X2/ATI HD 3200/RTL8191SEVA2 wlan0, RTL8169 eth0

Kringel
Posts: 44
Joined: 2012-10-20 10:33

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#4 Post by Kringel »

Ibidem wrote:Any reason for not going "make deb-pkg" (the upstream way) ?
The problem was that with "make deb-pkg" I found no way to add the version postfix. Let me know if you know how to do it. I would like to use it because make deb-pkg is the recommended method.
Ibidem wrote:Also, if you don't have any extra removable peripherals (ie, all the modules you'll use are loaded), make localmodconfig will really speed up the build (I think better than halve build time...).
But all in all, looks good.
Thank you. I added an option to use localmodconfig.

Ibidem
Posts: 160
Joined: 2010-12-24 18:28

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#5 Post by Ibidem »

I'd try "make deb-pkg CONFIG_LOCALVERSION=-upstream" or so (all the CONFIG_* variables in .config can be overridden on the make command line, afaict). I have not tried that yet.
Thinkpad X100e/Debian Squeeze (All reposiories enabled)/Linux 3.4.11:
1GB RAM/1.6GHz Neo X2/ATI HD 3200/RTL8191SEVA2 wlan0, RTL8169 eth0

User avatar
linearJim
Posts: 81
Joined: 2012-06-08 21:10

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#6 Post by linearJim »

hi
great post/script Kringel, i have a question though

inside the Build() function the line

Code: Select all

time fakeroot make-kpkg --jobs `getconf _NPROCESSORS_ONLN` --append-to-version "$VERSION_POSTFIX" --initrd kernel_image
creates only one deb file, the image one, so the new kernel cannot be installed

while if i add

Code: Select all

time fakeroot make-kpkg --jobs `getconf _NPROCESSORS_ONLN` --append-to-version "$VERSION_POSTFIX" --initrd kernel_image kernel_headers
it creates two deb files that matches your last echo command which says "install with dpkg -i *.deb" which assumes that more than one deb file is created, right?

anyway, when i compile the upstream kernel --with the kernel_headers addition-- the system hangs on gdm log in screen, but it works great if i compile (with makelocalmodconfig & make xconfig) my current kernel with $(uname -r) [2.3.32]
Οh and a lame crab walks straight

Kringel
Posts: 44
Joined: 2012-10-20 10:33

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#7 Post by Kringel »

The script currently creates exactly one .deb file (something like "linux-image-3.6.3-upstream_3.6.3-upstream-10.00.Custom_amd64.deb"). I wrote "*.deb" in the final message because I was to lazy to figure out what the full filename will look like. I do not expect the creation of multiple packages. So in that sense there is nothing wrong (apart from my laziness), you can install the kernel with that single .deb file.

To be honset I am not quite sure when the installation of a header package ist required. Also I am not sure what kernel versions can safely be installed (system components rely on a stable binary interface to the kernel... and at some day a kernel version will break this interface). At the end I ended up with a lazy solution again: I found some documentation that said the ABI-compatibility is checked when modules get loaded. So I expect an informative error message when something does not fit (when the new built kernel conflicts with your system). I just tried the kernel version that I needed... it worked and I stopped my research.

In fact I should figure out what targetes (headers, modules, ...) are requiered under what conditions (obviously for my case the single kernel image was sufficient). Unfortunately I found no satisfying documentation about this topic so far. man make-kpg does tell about the available targets, but does not say what is needed under what condition. Same for the links that I put as coments into the script.

It would be nice if anyone could point to good resources or/and explain the things in the two paragraphs above :wink:

To your hanging kernel: It seems obvious that this problem is caused by the different configuration files. I dont think that it matters if the headers are installed. Also "make localmodconfig" should just speed up the build (by omitting all modules that are not in use). The script uses the config of your running kernel and sets defaults for new options. I have no idea why this leads to a bad result.

User avatar
linearJim
Posts: 81
Joined: 2012-06-08 21:10

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#8 Post by linearJim »

To be honset I am not quite sure when the installation of a header package ist required.
Actually, whenever a there is a kernel upgrade (from the repos) for some reason only the image is installed and the system wont boot, each time i have to manually install the header files of the upgraded kernel in order to have a working system. from a random search ( http://blo.gr/9an ) i see that kernel-headers file is included in the make-kpkg command.

i tried to compile the kernel with the make deb-pkg (as mentioned above) and the documentation says

As a result of the build, custom kernel package linux-image-3.2.19_3.2.19-1_i386.deb (name will reflect the version of the kernel and build number) will be created in the directory one level above the top of the tree
i get 3 deb files

linux-headers-3.2.32_3.2.32-2_amd64.deb
linux-libc-dev_3.2.32-2_amd64.deb
linux-image-3.2.32_3.2.32-2_amd64.deb


confused :roll:
Οh and a lame crab walks straight

Kringel
Posts: 44
Joined: 2012-10-20 10:33

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#9 Post by Kringel »

For the headers I found a good explanation. So some things feel cleared up in my mind:

1) You dont have to worry about modules that are build together with the kernel, they will use the headers of that kernel (not the ones currently installed in the system).
2) You should worry about glibc (the system library that virtualy all programs depend on), as it is built against the kernel headers.
3) You also should worry about modules that are built later (out-of-tree modules), that will use the installed kernel headers.

About 2) I dont know what happens when the ABI of glibc and the kernel breaks (will programs just crash mysteriously?). If it breaks you will need the glibc package for the running kernel.

About 3) The documentation says "The kernel module loader detects incompatible modules by comparing symbol versions.". So you probably will get a usefull error message when you try to load a kernel modul that was built with an incompatible ABI (built with incompatible kernel headers). But the same documentation says "In order to avoid the need for users to rebuild out-of-tree modules frequently, we try to avoid changing the kernel ABI during updates to a Debian stable or oldstable release." So the ABI will stay stable as long as possible. But it can break with some newer/older kernel versions or different kernel confiugration.

So at the moment my strategy is to just try if my system runs with a new kernel. If it works I am happy. If not... I dont know.

I know I repeat myelf: If anyone knows the details of this stuff I would be happy to hear about it.

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#10 Post by stevepusser »

I would recommend the build and install of the header package. Without it, to build new kernel modules, such as ATI or Nvidia video drivers, or Virtual Box, require that you hang onto your folder of compiled kernel source, which will be over 1 GB in size, since it substitutes for the headers. It works, but at the cost of a lot of disk space. Plus it's a lot easier to share a header deb package among different machines.

Hey, your script works fine on Squeeze, but I did update kernel-package from Sid and initramfs-tools from squeeze-backports. I have a new laptop that even the 3.2 kernel could not control the CPU scaling, though it did get the Atheros 9485 wifi going along with suspend to RAM and sound. (AMD A6-4400M processor with ATI 7520G graphics) I used the smxi script to build the ATI driver without dropping out of the GUI, and contrary to what is said, it built and installed just fine without dropping out of X.

Edit: It seems for some reason, even if you build headers, the kernel install will still link the build and source folder in /lib/modules/kernel-version to the compiled source folder, though installing the headers and kernel on other machines will work correctly. It's easy enough to delete the links and create new ones that point to the header folder: (as root, and if you are running the new kernel)

cd /lib/modules/$(uname -r)
rm build
rm source
ln -s /usr/src/linux-headers-$(uname -r) source
ln -s /usr/src/linux-headers-$(uname -r) build
MX Linux packager and developer

User avatar
Soapm
Posts: 603
Joined: 2012-05-22 04:23
Has thanked: 1 time

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#11 Post by Soapm »

This is what I get. I get the same error when I type unxz and hit enter. Is there a prerequisite to make this script work or any instructions?

Code: Select all

--2013-06-11 21:15:13--  https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.8.tar.sign
Connecting to www.kernel.org (www.kernel.org)|149.20.4.69|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 490 [application/pgp-signature]
Saving to: `linux-3.8.tar.sign'

100%[======================================>] 490         --.-K/s   in 0.02s   

2013-06-11 21:15:13 (22.3 KB/s) - `linux-3.8.tar.sign' saved [490/490]

Extracting downloaded sources to tar...
[b]unxz: Compressed data cannot be read from a termina[/b]l

s_mcleod
Posts: 3
Joined: 2015-03-08 13:13

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#12 Post by s_mcleod »

Kringel wrote:The following shellscript will build a debian kernel package from upstream sources ...
Howdy all - I've been doing some work on CI builds of the Linux kernel for Debian.

I've taken @Kringel's great script, modified it a lot:

* Docker builds
* Public continuous integration builds
* Public apt repository
* Wrote a small script that watches kernel.org for new kernels and can perform actions such as update git repos, trigger builds etc...
* Automated builds generated as new stable kernels are released from the aforementioned script

Feel free to submit issues / pull requests etc... on Github.

Links:

* Github project: https://github.com/sammcj/kernel-ci
* Kernel Watch: https://github.com/sammcj/kernel-ci/blo ... l_watch.sh
* Travis builds: https://travis-ci.org/sammcj/kernel-ci
* Packagecloud packages: https://packagecloud.io/mrmondo/debian- ... ilter=debs
* Packagecloud apt-repo: deb https://packagecloud.io/mrmondo/debian-kernel/debian/ jessie main

Kringel
Posts: 44
Joined: 2012-10-20 10:33

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#13 Post by Kringel »

Nice to see this script is still in use!

Although you disabled the verification of the GPG signature and I don't see where the integrity of the downloaded sources is verified.

s_mcleod
Posts: 3
Joined: 2015-03-08 13:13

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#14 Post by s_mcleod »

Yes I'm well aware of that and it's logged as ticket #1 - https://github.com/sammcj/kernel-ci/issues/1
I found it to be quite unreliable, often it would fail for seemingly no reason - so I need to go back and work out what's going wrong.

Also I've just refactored a lot of the script - I've got a Travis run going now - it should get merged in later today:
https://travis-ci.org/sammcj/kernel-ci/builds/53596474
https://github.com/sammcj/kernel-ci/compare/cleanup

s_mcleod
Posts: 3
Joined: 2015-03-08 13:13

Re: HOWTO: Build an upstream kernel package (foolproof scrip

#15 Post by s_mcleod »

Kringel,

The GPG signature checking bug is now fixed.
I've also added in the option to add GRSecurity patches to the kernel before build and made it easy to specify kernel config changes easily.

https://github.com/sammcj/kernel-ci

Post Reply