Downloading buildd log for Debian packages with getbuildlog

[Aki]

Hello,

I'm using Debian Bookworm 12.5 amd64 (last updated today).

I'm trying to download the log generated by buildd (Debian build daemon) for the Debian package linux-signed-amd64 for Debian Bookworm.

These are the existing packages in Debian repositories for linux-signed-amd64:

Code: Select all

$ rmadison linux-signed-amd64
linux-signed-amd64 | 4.19.235+1         | oldoldstable       | source
linux-signed-amd64 | 4.19.249+2         | oldoldstable       | source
linux-signed-amd64 | 5.10.127+2~bpo10+1 | buster-backports   | source
linux-signed-amd64 | 5.10.197+1         | oldstable          | source
linux-signed-amd64 | 5.10.209+2         | oldstable          | source
linux-signed-amd64 | 6.1.55+1~bpo11+1   | bullseye-backports | source
linux-signed-amd64 | 6.1.66+1           | stable             | source
linux-signed-amd64 | 6.1.67+1           | stable-updates     | source
linux-signed-amd64 | 6.1.69+1~bpo11+1   | bullseye-backports | source
linux-signed-amd64 | 6.1.76+1~bpo11+1   | backports-policy   | source
linux-signed-amd64 | 6.1.76+1           | stable             | source
linux-signed-amd64 | 6.1.82+1           | proposed-updates   | source
linux-signed-amd64 | 6.1.85+1           | proposed-updates   | source
linux-signed-amd64 | 6.5.3+1~bpo12+1    | stable-backports   | source
linux-signed-amd64 | 6.5.10+1~bpo12+1   | stable-backports   | source
linux-signed-amd64 | 6.6.13+1~bpo12+1   | stable-backports   | source
linux-signed-amd64 | 6.6.15+2           | testing            | source
linux-signed-amd64 | 6.7.12+1           | buildd-unstable    | source
linux-signed-amd64 | 6.7.12+1           | unstable           | source

According to the "Debian Package Auto-Building" web page [1], I could use the getbuildlog [2] utility (from the devscripts package):

Download build logs

Install getbuildlog: sudo apt install devscripts
Download linux-signed-amd64 6.1.82+1 amd64 build log: getbuildlog linux-signed-amd64 6.1.82+1 amd64
Download all linux-signed-amd64 6.1.82+1 build logs: getbuildlog linux-signed-amd64 6.1.82+1
Download latest linux-signed-amd64 amd64 build log: getbuildlog linux-signed-amd64 "last" amd64
Download all linux-signed-amd64 amd64 build logs: getbuildlog linux-signed-amd64 "" amd64
Download latest linux-signed-amd64 build logs: getbuildlog linux-signed-amd64 last
Download all linux-signed-amd64 build logs: getbuildlog linux-signed-amd64

The following command successfully downloads all build logs:

Code: Select all

$ getbuildlog linux-signed-amd64 

On the contrary, the following command do not download the buildd log for the 6.1.82+1 version:

Code: Select all

$ getbuildlog linux-signed-amd64 6.1.82+1 amd64

Analysing the script running, this is its output:

Code: Select all

$ bash -x getbuildlog linux-signed-amd64 6.1.82+1 amd64
+ set -e
+ PROGNAME=getbuildlog
+ '[' linux-signed-amd64 = -h ']'
+ '[' linux-signed-amd64 = --help ']'
+ '[' linux-signed-amd64 = -V ']'
+ '[' linux-signed-amd64 = --version ']'
+ '[' 3 -ge 1 ']'
+ '[' 3 -le 3 ']'
+ which wget
+ PACKAGE=linux-signed-amd64
+ VERSION=6.1.82+1
+ ARCH=amd64
++ echo linux-signed-amd64
++ sed -e 's/\+/\\\+/g'
+ ESCAPED_PACKAGE=linux-signed-amd64
+ GET_LAST_VERSION=no
+ '[' 6.1.82+1 = last ']'
+ '[' 6.1.82+1 = last-all ']'
+ PATTERN='fetch\.(cgi|php)\?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82+1&stamp=[[:digit:]]+'
+ getbuildlog https://buildd.debian.org
+ BASE=https://buildd.debian.org
++ mktemp --tmpdir getbuildlog.tmp.XXXXXXXXXX
+ ALL_LOGS=/tmp/getbuildlog.tmp.u6G4nlCLNz
+ wget -q -O /tmp/getbuildlog.tmp.u6G4nlCLNz 'https://buildd.debian.org/status/logs.php?pkg=linux-signed-amd64'
+ sed -i -e 's/href="/\nhref="/g' /tmp/getbuildlog.tmp.u6G4nlCLNz
+ sed -i -e 's/&/\&/g' -e 's/%2B/\+/g' -e s/%3A/:/g -e s/%7E/~/g /tmp/getbuildlog.tmp.u6G4nlCLNz
+ '[' no '!=' no ']'
+ NEWPATTERN='fetch\.(cgi|php)\?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82+1&stamp=[[:digit:]]+'
++ grep -E -o 'fetch\.(cgi|php)\?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82+1&stamp=[[:digit:]]+' /tmp/getbuildlog.tmp.u6G4nlCLNz
+ '[' no = yes ']'

Therefore, the following regex expression does not match any lines in the list of buildd logs available at https://buildd.debian.org/status/logs.php?pkg=linux-signed-amd64:

Code: Select all

PATTERN='fetch\.(cgi|php)\?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82+1&stamp=[[:digit:]]+'
[..]
grep -E -o 'fetch\.(cgi|php)\?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82+1&stamp=[[:digit:]]+' /tmp/getbuildlog.tmp.u6G4nlCLNz
[..]

It turns out that the regex fails because of the '+' character (which is interpreted as a regex expression, not a character); the regex expression that matches the log name is (note the double back slash to escape the '+' character):

Code: Select all

grep -E -o 'fetch\.(cgi|php)\?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82\\+1&stamp=[[:digit:]]+' /tmp/getbuildlog.tmp.u6G4nlCLNz

Therefore, the command should be:

Code: Select all

$ getbuildlog linux-signed-amd64 6.1.82\\+1 amd64
--2024-04-26 15:52:45--  https://buildd.debian.org/status/fetch.php?pkg=linux-signed-amd64&arch=amd64&ver=6.1.82%2B1&stamp=1712000451&raw=1
Resolving buildd.debian.org (buildd.debian.org)... 209.87.16.60, 2607:f8f0:614:1::1274:60
Connecting to buildd.debian.org (buildd.debian.org)|209.87.16.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: 'linux-signed-amd64_6.1.82+1_amd64.log'

linux-signed-amd64_6.1.82+1_amd     [      <=>                                            ]   1.81M  1.59MB/s    in 1.1s    

2024-04-26 15:52:48 (1.59 MB/s) - 'linux-signed-amd64_6.1.82+1_amd64.log' saved [1894891]

or more precisely:

Code: Select all

getbuildlog linux-signed-amd64 6\\.1\\.82\\+1 amd64

Am I missing something ?

What could be changed in the getbuildlog script [2] to help fix this and contribute to the Debian project?

--
[1] Debian Package Auto-Building
[2] getbuildlog.sh

[fabien]

It turns out that the regex fails because of the '+' character (which is interpreted as a regex expression, not a character)

It looks like a feature, not a bug

Code: Select all

$> getbuildlog -h
[...]
Examples:
  # Download amd64 build log for hello version 2.2-1:
  getbuildlog hello 2\.2-1 amd64

We see that the help shows that the patterns must be protected.
This allows something like 6\.1\.6+\+1 for example.

Code: Select all

$> getbuildlog linux-signed-amd64 "6\.1\.6+\+1" amd64
--2024-04-26 23:40:46--  https://buildd.debian.org/status/fetch.php?pkg=linux-signed-amd64&arch=amd64&ver=6.1.66%2B1&stamp=1702199523&raw=1
Resolving buildd.debian.org (buildd.debian.org)... 209.87.16.60, 2607:f8f0:614:1::1274:60
Connecting to buildd.debian.org (buildd.debian.org)|209.87.16.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/plain]
Saving to: 'linux-signed-amd64_6.1.66+1_amd64.log'

linux-signed-amd64_6.1.66+1_amd64.log                   [        <=>                                                                                                       ]   1.76M  1.08MB/s    in 1.6s    

2024-04-26 23:40:48 (1.08 MB/s) - 'linux-signed-amd64_6.1.66+1_amd64.log' saved [1846625]

What I think is a bug is on line 76:

Code: Select all

ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\+/\\\+/g')

"." must also be protected in package names.

Since it's a shell script, it's easy to have a modified version in /usr/local/bin/.
I can propose "protected-patterns-getbuildlog" which keeps the default behavior by default but automatically protects version patterns when using the -p switch.

Code: Select all

#!/bin/sh
#
# getbuildlog: download package build logs from Debian auto-builders
#
# Copyright © 2008 Frank S. Thomas <fst@debian.org>
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

set -e

PROGNAME=${0##*/}

usage() {
    cat <<EOT
Usage: $PROGNAME <package> [<version-pattern>] [<architecture-pattern>]
  Downloads build logs of <package> from Debian auto-builders.
  If <version-pattern> or <architecture-pattern> are given, only build logs
  whose versions and architectures, respectively, matches the given patterns
  are downloaded.

  If <version-pattern> is "last" then only the logs for the most recent
  version of <package> found on buildd.debian.org will be downloaded.

  If <version-pattern> is "last-all" then the logs for the most recent
  version found on each build log index will be downloaded.
Options:
  -h, --help        Show this help message.
  -V, --version     Show version and copyright information.
Examples:
  # Download amd64 build log for hello version 2.2-1:
  $PROGNAME hello 2\.2-1 amd64

  # Same as above but automatically protect version:
  $PROGNAME -p hello 2.2-1 amd64

  # Download mips(el) build logs of all glibc versions:
  $PROGNAME glibc "" mips.*

  # Download all build logs of backported wesnoth versions:
  $PROGNAME wesnoth .*bpo.*
EOT
}

version() {
    cat <<EOT
This is $PROGNAME, from the Debian devscripts package, version 2.23.4+deb12u1
This code is copyright 2008 by Frank S. Thomas, all rights reserved.
This program comes with ABSOLUTELY NO WARRANTY.
You are free to redistribute this code under the terms of the
GNU General Public License, version 2 or later.
This version 2024 of1_À_disroot_POINT_org GPLv3+
EOT
}

### -p automatically protects versions (treated as patterns by default)
[ "$1" = "-p" ] && { shift; PROTECTEDPTRN="true"; } || PROTECTEDPTRN="false"

[ "$1" = "-h" ] || [ "$1" = "--help" ] && usage && exit 0
[ "$1" = "-V" ] || [ "$1" = "--version" ] && version && exit 0

[ $# -ge 1 ] && [ $# -le 3 ] || { usage && exit 1; }

if ! which wget > /dev/null 2>&1; then
    echo "$PROGNAME: this program requires the wget package to be installed";
    exit 1
fi

PACKAGE=$1
VERSION=${2:-[:~+.[:alnum:]-]+}
ARCH=${3:-[[:alnum:]-]+}
### protect + AND .
ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\.\|\+/\\&/g')

GET_LAST_VERSION=no
if [ "$VERSION" = "last" ]; then
    GET_LAST_VERSION=yes
    VERSION=[:~+.[:alnum:]-]+
elif [ "$VERSION" = "last-all" ]; then
    GET_LAST_VERSION=all
    VERSION=[:~+.[:alnum:]-]+
elif "$PROTECTEDPTRN" && [ "$VERSION" != "[:~+.[:alnum:]-]+" ]; then
    VERSION=$(echo "$VERSION" | sed -e 's/\.\|\+/\\&/g')
fi

PATTERN="fetch\.(cgi|php)\?pkg=$ESCAPED_PACKAGE&arch=$ARCH&ver=$VERSION&\
stamp=[[:digit:]]+"

getbuildlog() {
    BASE=$1
    ALL_LOGS=$(mktemp --tmpdir getbuildlog.tmp.XXXXXXXXXX)

    trap 'rm -f "$ALL_LOGS"' EXIT

    wget -q -O $ALL_LOGS "$BASE/status/logs.php?pkg=$PACKAGE"

    # Put each href in $ALL_LOGS on a separate line so that $PATTERN
    # matches only one href. This is required because grep is greedy.
    sed -i -e "s/href=\"/\nhref=\"/g" $ALL_LOGS
    # Quick-and-dirty unescaping
    sed -i -e "s/&amp;/\&/g" -e "s/%2B/\+/g" -e "s/%3A/:/g" -e "s/%7E/~/g" $ALL_LOGS

    # If only the last version was requested, extract and sort
    # the listed versions and determine the highest
    if [ "$GET_LAST_VERSION" != "no" ]; then
	LASTVERSION=$( \
	    for match in $(grep -E -o "$PATTERN" $ALL_LOGS); do
		ver=${match##*ver=}
		echo ${ver%%&*}
	    done | perl -e '
		use Devscripts::Versort;
		while (<>) { push @versions, [$_]; }
		@versions = Devscripts::Versort::versort(@versions);
		print $versions[0][0]; ' | sed -e "s/\+/\\\+/g"
	)

	NEWPATTERN="fetch\.(cgi|php)\?pkg=$ESCAPED_PACKAGE&\
arch=$ARCH&ver=$LASTVERSION&stamp=[[:digit:]]+"
    else
	NEWPATTERN=$PATTERN
    fi

    for match in $(grep -E -o "$NEWPATTERN" $ALL_LOGS); do
	ver=${match##*ver=}
	ver=${ver%%&*}
	arch=${match##*arch=}
	arch=${arch%%&*}
	match=$(echo $match | sed -e 's/\+/%2B/g')
	# Mimic wget's behaviour, using a numerical suffix if needed,
	# to support downloading several logs for a given tuple
	# (unfortunately, -nc and -O means only the first file gets
	# downloaded):
	filename="${PACKAGE}_${ver}_${arch}.log"
	if [ -f "$filename" ]; then
	    suffix=1
	    while [ -f "$filename.$suffix" ]; do suffix=$((suffix+1)); done
	    filename="$filename.$suffix"
	fi
	wget -O "$filename" "$BASE/status/$match&raw=1"
    done

    rm -f $ALL_LOGS

    if [ "$GET_LAST_VERSION" = "yes" ]; then
	PATTERN=$NEWPATTERN
	GET_LAST_VERSION=no
    fi
}

getbuildlog https://buildd.debian.org

Code: Select all

$> diff /usr/bin/getbuildlog /usr/local/bin/pp-getbuildlog 
44a45,47
>   # Same as above but automatically protect version:
>   $PROGNAME -p hello 2.2-1 amd64
> 
59a63
> This version 2024 of1_À_disroot_POINT_org GPLv3+
62a67,69
> ### -p automatically protects versions (treated as patterns by default)
> [ "$1" = "-p" ] && { shift; PROTECTEDPTRN="true"; } || PROTECTEDPTRN="false"
> 
76c83,84
< ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\+/\\\+/g')
---
> ### protect + AND .
> ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\.\|\+/\\&/g')
84a93,94
> elif "$PROTECTEDPTRN" && [ "$VERSION" != "[:~+.[:alnum:]-]+" ]; then
>     VERSION=$(echo "$VERSION" | sed -e 's/\.\|\+/\\&/g')

[Aki]

Hello @fabien,

I apologize for not replaying in a short time.

What I think is a bug is on line 76:

Code: Select all

ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\+/\\\+/g')

"." must also be protected in package names.
Since it's a shell script, it's easy to have a modified version in /usr/local/bin/.
I can propose "protected-patterns-getbuildlog" which keeps the default behaviour by default but automatically protects version patterns when using the -p switch.
[..]

Code: Select all

$> diff /usr/bin/getbuildlog /usr/local/bin/pp-getbuildlog 
44a45,47
>   # Same as above but automatically protect version:
>   $PROGNAME -p hello 2.2-1 amd64
> 
59a63
> This version 2024 of1_À_disroot_POINT_org GPLv3+
62a67,69
> ### -p automatically protects versions (treated as patterns by default)
> [ "$1" = "-p" ] && { shift; PROTECTEDPTRN="true"; } || PROTECTEDPTRN="false"
> 
76c83,84
< ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\+/\\\+/g')
---
> ### protect + AND .
> ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\.\|\+/\\&/g')
84a93,94
> elif "$PROTECTEDPTRN" && [ "$VERSION" != "[:~+.[:alnum:]-]+" ]; then
>     VERSION=$(echo "$VERSION" | sed -e 's/\.\|\+/\\&/g')

Thank you so much for your insights.

If I understand correctly, line 76 only affects the name of the package (escaping the "+" sign), but not the "+" or "." characters in the package's version string:

Code: Select all

ESCAPED_PACKAGE=$(echo "$PACKAGE" | sed -e 's/\+/\\\+/g')

Following your insights, the original getbuildlog script fails because of the package's version argument: the "." and "+" are not escaped, therefore in the script's regex the first is interpreted as a "metacharacter", the second as a "quantifier", both not as character); i.e. the following commands fail:

Code: Select all

$ getbuildlog linux-signed-amd64  6.1.67+1
$ getbuildlog linux-signed-amd64 "6.1.67+1" 

Otherwise, the following command succeeds because in the regular expression "6.1.67\+1" the character "." is interpreted as metacharacter (that matches every other character; note: possible side effects), but "\+" as a character:

Code: Select all

$ getbuildlog linux-signed-amd64 "6.1.67\+1"

Therefore, the complete command working command with the original script should be:

Code: Select all

$ getbuildlog linux-signed-amd64 "6\.1\.67\+1"

So I thought another possible change to the original script might be (full script attached):

Code: Select all

$ diff -u /bin/getbuildlog my_getbuildlog.sh
--- /bin/getbuildlog    2023-10-20 18:54:42.000000000 +0200
+++ my_getbuildlog.sh   2024-05-01 00:53:21.419765425 +0200
@@ -82,8 +82,11 @@
 elif [ "$VERSION" = "last-all" ]; then
     GET_LAST_VERSION=all
     VERSION=[:~+.[:alnum:]-]+
+else
+    VERSION=$(echo "$VERSION" | sed 's/\.\|\+/\\&/g')
 fi
 
 PATTERN="fetch\.(cgi|php)\?pkg=$ESCAPED_PACKAGE&arch=$ARCH&ver=$VERSION&\
 stamp=[[:digit:]]+"

This way the script works also without quoting or escaping "." and "+" characters; i.e.:

Code: Select all

$ bash ./my_getbuildlog.sh linux-signed-amd64 6.1.67+1

Therefore command format for version argument could be compatible with the commands cited in the footer of Debian buildd page [1], but the manual page of the command should be modified.

I cannot see now if there are other side effects.

What do you think about it ?

---

my_getbuildlog.sh.zip

(2.27 KiB) Downloaded 7 times

--
[1] buildd page

[fabien]

I apologize for not replaying in a short time.

You're welcome

This way the script works also without quoting or escaping "." and "+" characters; i.e.:

Code: Select all

$ bash ./my_getbuildlog.sh linux-signed-amd64 6.1.67+1

Therefore command format for version argument could be compatible with the commands cited in the footer of Debian buildd page [1], but the manual page of the command should be modified.

I cannot see now if there are other side effects.

What do you think about it ?

I now see what the root of your problem is, that the pages on Debian Package Auto-Building have this little flaw. One possibility might be to contact the person responsible for these pages to ask if this flaw in the instructions could be corrected. The reason being that if you change getbuildlog the pages will only be correct for those using the new version, if you fix the pages the pages will immediately be correct for everyone. That's why I proposed a less disruptive approach that, by default, retains traditional behaviour. Changing behaviour is a concern for developers because it can break user processes and, to my knowledge, they usually don't do it lightly (at least that's the theory).
I suppose the developer had a good reason to treat this argument as a pattern, even if this feature is not useful to everyone. This allows commands like getbuildlog linux-signed-amd64 "6\.1.*" or getbuildlog linux-signed-amd64 "6\.1.+", I doubt they're willing to give that up

[Aki]

Hello,

I now see what the root of your problem is, that the pages on Debian Package Auto-Building have this little flaw. One possibility might be to contact the person responsible for these pages to ask if this flaw in the instructions could be corrected. The reason being that if you change getbuildlog the pages will only be correct for those using the new version, if you fix the pages the pages will immediately be correct for everyone. That's why I proposed a less disruptive approach that, by default, retains traditional behaviour. Changing behaviour is a concern for developers because it can break user processes and, to my knowledge, they usually don't do it lightly (at least that's the theory).
I suppose the developer had a good reason to treat this argument as a pattern, even if this feature is not useful to everyone. This allows commands like getbuildlog linux-signed-amd64 "6\.1.*" or getbuildlog linux-signed-amd64 "6\.1.+", I doubt they're willing to give that up :)

Thanks, that's a good point.