Scheduled Maintenance: We are aware of an issue with Google, AOL, and Yahoo services as email providers which are blocking new registrations. We are trying to fix the issue and we have several internal and external support tickets in process to resolve the issue. Please see: viewtopic.php?t=158230

 

 

 

HOWTO: Backport and host Debian & Ubuntu packages on the OBS

Share your HowTo, Documentation, Tips and Tricks. Not for support questions!.
Post Reply
Message
Author
User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

HOWTO: Backport and host Debian & Ubuntu packages on the OBS

#1 Post by stevepusser »

* NOTE: as explained below, more and more backports are requiring a newer debhelper
and dh-autoreconf in each repository (project) we create. We can cut down on the
work by backporting everything we do in one project, so they all use the backported
debhelper and dh-autoreconf we've already added to the project.

The alternative is to just remove the need for those newer build tools. This can usually
be done by commenting out the two lines in the override section for dh_strip in /debian/rules,
and dropping the minimum required version of the debhelper build-depend in the debian/control
and compat files if necessary (often the maintainer forgets to bump the B-D) We have the
choice--whatever's the least trouble.

Note 2: Sometimes the OBS has trouble with build-depends that depend on a choice of two other build-depends, such as libjpeg-dev and libjpeg62-turbo-dev. ( Hover the mouse over the "unresovable" error to see what's going on) In this particular case, just add libjpeg-dev to the build-depends in debian/control--or whatever "general" build-depend the mouse hover reveals, then rebuild the sources again, delete the debian.tar.xz and .dsc file from the OBS, and upload your new versions of those.

Note 3: The OBS now seems to support the "+" symbols in package versions. I've tried to update the guide to cover this, but keep this in mind if I missed anything.

(Part 1 of 3)

OBS = openSUSE Build Service: https://build.opensuse.org/

The free and easy to join openSUSE Build Service builds and hosts both Linux rpm
and deb packages from source packages. This guide focuses on building packages
for Debian and Ubuntu (deb format).

Prerequisites:
1. An account at the OBS, which is easy to create via their web
interface. Once we're logged in, currently a cookie will keep us logged in for 24
hours, or until we log out.

2. Have the "devscripts" package installed on our machine. You may also need to have
certain of the build-depends installed on your machine to rebuild the source files.

3. Source packages to load into the OBS. If you're now not interested in doing a
smooth in-place upgrade to the next Debian release with your backport installed,
you could try directly downloading and installing the source packages from
Debian, though often you can tweak those to use debhelper 10 in Stretch instead
of Buster's debhelper 11, or other alternative build-depends. Then you need to
rebuild the sources as explained below. Smooth upgrades also require changing
the version as explained.

Source packages can be obtained from upstream Debian, Ubuntu, PPAs, or other
repositories as described here: https://wiki.debian.org/SimpleBackportCreation
If we don't plan to actually build any packages on our own machine, we should
be able to get by with just an install of devscripts instead of the bigger
packaging-dev metapackage. Source packages can also be directly downloaded to
our machine from packages.debian.org,

Image

or web surfing down into the package pool in a PPA, for example. This can be quicker
than modifying our software sources, downloading, then reverting the changes.

Often, the upstream Debian source packages can be directly uploaded into the OBS
without changes. However, if we, like the official backports crew, are concerned
with allowing users a smooth upgrade in place to the next Debian release, adding a
tilde "~" symbol allows our version to be seen as a lower version than the same
one in the new stable. So if the upstream version is

foo-1.2.3-1


at the time of the stable transition, changing that to

foo 1.2.3-1~obs

will make that a lower version than the first, and the user will be able to upgrade
smoothly to the new Debian stable version.
J̶e̶s̶s̶i̶e̶-̶b̶a̶c̶k̶p̶o̶r̶t̶s̶ ̶a̶d̶d̶s̶ ̶"̶~̶b̶p̶o̶8̶0̶+̶1̶,̶ ̶a̶n̶d̶ ̶o̶r̶d̶i̶n̶a̶r̶i̶l̶y̶ ̶w̶e̶'̶d̶ ̶b̶e̶ ̶a̶b̶l̶e̶ ̶t̶o̶ ̶u̶p̶l̶o̶a̶d̶ ̶t̶h̶o̶s̶e̶ ̶s̶o̶u̶r̶c̶e̶ ̶p̶a̶c̶k̶a̶g̶e̶s̶ ̶d̶i̶r̶e̶c̶t̶l̶y̶,̶ ̶b̶u̶t̶ ̶u̶n̶f̶o̶r̶t̶u̶n̶a̶t̶e̶l̶y̶,̶ ̶t̶h̶e̶ ̶O̶B̶S̶ ̶d̶o̶e̶s̶ ̶n̶o̶t̶ ̶r̶e̶c̶o̶g̶n̶i̶z̶e̶ ̶t̶h̶e̶ ̶"̶+̶"̶ ̶s̶y̶m̶b̶o̶l̶,̶ ̶s̶o̶ ̶w̶e̶ ̶h̶a̶v̶e̶ ̶t̶o̶ ̶e̶i̶t̶h̶e̶r̶ ̶a̶d̶d̶ ̶a̶ ̶n̶e̶w̶ ̶d̶e̶b̶i̶a̶n̶/̶c̶h̶a̶n̶g̶e̶l̶o̶g̶ ̶s̶t̶a̶n̶z̶a̶ ̶w̶i̶t̶h̶ ̶a̶ ̶m̶o̶d̶i̶f̶i̶e̶d̶ ̶v̶e̶r̶s̶i̶o̶n̶ ̶(̶~̶b̶p̶o̶8̶0̶.̶1̶)̶,̶ ̶p̶e̶r̶ ̶t̶h̶e̶ ̶w̶i̶k̶i̶ ̶p̶r̶o̶c̶e̶s̶s̶ ̶a̶b̶o̶v̶e̶,̶ ̶o̶r̶ ̶d̶i̶r̶e̶c̶t̶l̶y̶ ̶e̶d̶i̶t̶ ̶t̶h̶e̶ ̶v̶e̶r̶s̶i̶o̶n̶ ̶s̶t̶r̶i̶n̶g̶ ̶i̶n̶ ̶t̶h̶e̶ ̶e̶x̶i̶s̶t̶i̶n̶g̶ ̶f̶i̶r̶s̶t̶ ̶s̶t̶a̶n̶z̶a̶.̶
Update: the OBS now allows the "+" symbol. I'll do that below in my example--what's important is the "~" to allow the upgrade process. If we aren't concerned about a smooth upgrade, perhaps if the
package isn't in upstream Debian and probably won't be in the next release, then
this isn't a problem.

The OBS can build for multiple Debian and Ubuntu releases from one set of source
packages, such as in my Pale Moon repository, so in that case, the version
shouldn't be distro-specific. Since PM isn't in those distros, I just use a version
like "26.5.0-1" in the changelog. Having distro-specific versioning, such as
"26.5.0-1~bpo9+1", would require having a separate subproject repository (see below)
for each distro with its own specific source packages, plus a single distro build
target for each subproject. That's not a problem if we're just doing backports
for one release, of course.

If your build requires a lot of RAM or disk space, such as for web browsers,
Debian kernels, or llvm backports, you can request the OBS supply build machines
with those by also uploading a "_constraints" file with the sources. Here's one I use
for Pale Moon:

Code: Select all

<?xml version="1.0"?>
<constraints>
  <hardware>
    <disk>
        <size unit="G">30</size>
    </disk>
    <memory>
      <size unit="M">10000</size>
    </memory>
  </hardware>
</constraints>


The OBS will also trigger a rebuild of everything in a repository if you upgrade
one of the base build-dependencies. For example, I have a multimedia repository in
which most of the packages depend on the latest ffmpeg release. Upgrading ffmpeg
also ends up triggering a rebuild of most packages. Upgrading an ffmpeg build-
depend such as libx265-dev also triggers a ffmpeg rebuild and the rest of the
cascade. Though this might take a few hours to complete, this is actually more
of a feature, since it ensures that all those multimedia packages remain compatible
across soname bumps, which can cause breakages deb-multimedia and even jessie-
backports have had trouble avoiding.

So, we created a new account on the OBS. We automatically get assigned a "home
project". For my example below, since I already have Pale Moon in my home project's
repository, I'll create a separate subproject so the repositories remain separate.
So, since we'll modifying the source, and this is a modern 3.0 (quilt) source
format package, we only need to download the two highlighted files shown in the
previous image. Then the two tarballs are extracted:

Image

and the version in the first stanza of the /debian/changelog file is modified,
per the wiki backporting guide. If we're really following Debian policy exactly,
we go through the procedure to add a new stanza, but for the OBS, we can get away
with just adding something like ~bpo9~obs to the existing version. In this case,
I already added a new stanza for the MX 15 backport, so I'm just going to edit that:

Image

Image

As before, the "~" is often the most important symbol. If you don't care about that,
just try downloading and uploading all the Debian source packages (usually three).
Also note that the build tools are really picky about syntax in this file, so if
we accidentally delete one of two whitespaces between words in certain places in
this file, the build system can throw a fit. Other places, it doesn't care. Safest to
just not change them, just carefully change words.

(continued)
Last edited by stevepusser on 2018-12-20 20:28, edited 7 times in total.
MX Linux packager and developer

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#2 Post by stevepusser »

(Part 2 of 3)

So once our changes are saved to the changelog, we put the /debian folder into the
extracted source folder, then enter the source folder, open a terminal there, and
issue:

Code: Select all

debuild -S -uc -us -d
Image

This regenerates a new debian tarball and a .dsc digital hash file. Since we changed
the contents of /debian, the hashes changed, we need new files. The orig tarball
should never change, so if it's a big one, we could be uploading it to the OBS to
save some time while we work on the others.

Anyway, here's my home project page, already occupied by Pale Moon. I don't want
any users of that repo to be confused with a new Gnumeric, so I'll go to my Subprojects
and create a new one for Gnumeric.

Image

Image

So I hit the (3) link for a new one.

Give it a name--Title and Description are optional, and hit "Create Project".

Image

Success!

Image

Now we need to add target repositories for our project, so hit the Repositories
link. In this screen, checking the box for a repository will automatically add it,
and we'll get a success message at the top. We're just going to add Jessie; repositories
can also be added and deleted later.

We can navigate around by clicking on the breadcrumb set of links, most of which
are visible at the top of the above graphic, so clicking on "Gnumeric" takes us to the
subproject's "root".

Image

Our source files get uploaded into a package, so we have to create one for
gnumeric. It's important that we use the actual name of a package that gets built
as listed in the /debian/control file. This makes it easy to publish our repository.
We only need one actual package name that will appear in our repo to publish it,
though--so gnumeric is good enough, and we could be more lax with the other four
packages it turns out we need to add later. So projects can have any name, but we'll
need to make sure one actual package name exists in that project.

Creating a gnumeric package, again anything besides a name is nice but optional:

Image

Success:

Image

(continued)
Last edited by stevepusser on 2018-01-26 21:08, edited 2 times in total.
MX Linux packager and developer

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#3 Post by stevepusser »

Part 3 of 3

Now we need to upload our source files for the package, so click on "Add file",
then choose a local file (the orig.tar.xz for gnumeric in this case)

Image

and "Save Changes" to upload it. Chromium-based browsers can display a % progress of
the file upload at the bottom, I haven't seen it in Firefox, though.

Image

Once uploaded, eventually OBS will display success and allow the next source file
to be uploaded.

Image

Once all are loaded, we give OBS a minute or two, then refresh the page to see the
build status.

Image

Oh, no! Standard Jessie is missing something needed to build this package. Mouse
over the status message on the right to see what.

Image

So we also need to backport newer versions of libgoffice-0.10-dev and libgsf-1-dev,
using the same procedure, and load their sources into new packages for our project.
Then the gnumeric build will recognize and use the new versions once they are complete.

So one of those new builds fails eventually anyway. Clicking on the "failed"
message brings up the build log with the error:

Image

This means that we need the debhelper from jessie-backports, and that the Debian
maintainer forgot to list the new minimum required version in the build-depends.
Another unresolvable then shows we need dh-autoreconf 12 from backports, also.
So we get the tar.xz source files for each from jessie-backports (just one for each,
they're native Debian packages, so include a debian folder already) Ordinarily, we
could just upload their tarballs and dsc file to the OBS, but we have to get rid of the
"+" in their "~bpo80+1" version string and make it "bpo80.1", again in the first
changelog stanza, and then regenerate new source files. In this case, we have to
upload the new tar.xz and .dsc file we made for each package. If you plan any more
backport projects on the OBS, keep the sources for these two--you'll need them.

If the build fails, and it can be fixed by editing a file in the /debian folder,
go ahead and make the changes, then regenerate the source files with that debuild
command. The hashes for the orig.tar file won't change, so it won't need to be
reuploaded. The package version also won't need to change if we haven't published
the packages (OK, that's impossible if the build failed), but it should be bumped
up to appear as an upgrade if others possibly have installed successful builds, and
we changed something--maybe added a feature. For a full version upgrade, such as
from 1.12.32 to 1.12.33, then all source files must be upgraded.

To delete a source file, just click on that red dot to the right of the source file
to bring up a deletion dialog.

After everything, success.

Image

Note that I didn't bother to give the newer debhelper or dh-autoreconf packages
the proper names, though anyone that adds our new repository will see the newer correct
versions from the debian/control file. They won't need to upgrade those two, though--
those are only build tools. Upgrading gnumeric will automatically pull in the newer
gsf and goffice packages, though.

Click on the gnumeric package link, since that's the one with the correct package
name.

Image

The repository is automatically created and a key created. We just need to give out
the "Download package" link, which has the instructions for adding the repo and the
key.
Last edited by stevepusser on 2016-10-06 17:08, edited 1 time in total.
MX Linux packager and developer

kaddy
Posts: 8
Joined: 2010-10-17 07:34

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#4 Post by kaddy »

Thanks for doing this... seems easy to follow and clear :D

I'll give it a go in a day or so and report back :)

User avatar
riteshsarraf
Posts: 6
Joined: 2004-09-15 04:44
Location: India
Contact:

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#5 Post by riteshsarraf »

Don't they allow simply adding the Debian Backports APT repo into it ?

I can see if the outdated packages is huge, it is going to be a cumbersome process keeping track of each dependnecy.

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#6 Post by stevepusser »

There very well might be that capability, but there's a lot of things about the OBS I don't know how to use. If you know how to do that, please let us know!
MX Linux packager and developer

User avatar
riteshsarraf
Posts: 6
Joined: 2004-09-15 04:44
Location: India
Contact:

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#7 Post by riteshsarraf »

stevepusser wrote:There very well might be that capability, but there's a lot of things about the OBS I don't know how to use. If you know how to do that, please let us know!

OBS does not seem to offer Debian + Backports. This is problematic when one wants to build for, say, Raspbian Jessie.
The other, unresolved problem I'm having right now is, that all Debian_8 builds are marked "excluded". And with the limited documentation that OBS has, it is not helpful. You can see my attempt here.

https://build.opensuse.org/package/show ... :kodi/kodi

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#8 Post by stevepusser »

Hmmm....I got Kodi 17.1 to build in my own backports multimedia repo by reusing the source packages from Debian Experimental, but it wasn't a simple procedure. That version requires quite a few backports that are already in that repo, notably ffmpeg and all the backports it needs, or for certain build-depends or depends, an alternate Jessie package can be used instead. I also had to change the versioning to make the "+" symbol a ".", then rename the source packages to reflect that.

The changed source packages (six altogether) are in here: http://download.opensuse.org/repositori ... ebian_8.0/

The multimedia backports project is https://build.opensuse.org/project/show ... evepassert

The good news is that I know it's possible for you to add my multimedia backports as a source for your own project, since I tried to delete a Qt 5.5 backport project I had done and then mangled by trying to upgrade it to 5.6, but the OBS wouldn't let me do it since someone else had added it as a source. The bad news is that I'm not really sure how to go about adding an outside OBS repo. Also I only have i386 and amd64 architectures...did you have to send an email to get all those other architectures enabled? If so, I'd like to know how to do that so I could also add a few of those arches, too.
MX Linux packager and developer

User avatar
riteshsarraf
Posts: 6
Joined: 2004-09-15 04:44
Location: India
Contact:

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#9 Post by riteshsarraf »

stevepusser wrote:Hmmm....I got Kodi 17.1 to build in my own backports multimedia repo by reusing the source packages from Debian Experimental, but it wasn't a simple procedure. That version requires quite a few backports that are already in that repo, notably ffmpeg and all the backports it needs, or for certain build-depends or depends, an alternate Jessie package can be used instead. I also had to change the versioning to make the "+" symbol a ".", then rename the source packages to reflect that.
This is fantastic. My intent is to just get Kodi packages for Raspbian Jessie. I don't think OBS currently has an offering for Backports, so I cut down the build to exclude packages that needed a newer version from Backports. This was mostly related to libcec, libp8-platform and one more package. Those features are something I don't really use much in my setup.
stevepusser wrote: The changed source packages (six altogether) are in here: http://download.opensuse.org/repositori ... ebian_8.0/

The multimedia backports project is https://build.opensuse.org/project/show ... evepassert
Hey Steve, Are you a DD or part of the Debian Multimedia Team ?
stevepusser wrote:The good news is that I know it's possible for you to add my multimedia backports as a source for your own project, since I tried to delete a Qt 5.5 backport project I had done and then mangled by trying to upgrade it to 5.6, but the OBS wouldn't let me do it since someone else had added it as a source. The bad news is that I'm not really sure how to go about adding an outside OBS repo. Also I only have i386 and amd64 architectures...did you have to send an email to get all those other architectures enabled? If so, I'd like to know how to do that so I could also add a few of those arches, too.
Yes. I'd love to rather collaborate with you and we leverage each others skill and time on this one. Like I mentioned above, I want to have regular builds for Kodi for Raspbian, something I don't see offered by anyone officially, yet. The natural and easier choice would have been Launchpad, if they had made an offering for Debian too. But LP being LP.

Let me run through the ARCH enablement steps. I'll post it here again.

User avatar
riteshsarraf
Posts: 6
Joined: 2004-09-15 04:44
Location: India
Contact:

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#10 Post by riteshsarraf »

For enabling more architectures, please try the following.

* Go to your project page. Eg. https://build.opensuse.org/project/show ... teshsarraf
* Then go to "Repositories"
* Then click on "Edit Repositories" for your Debian_8 repository.
* There you should have an option to select your architectures, like in the below screenshot.

https://goo.gl/photos/Cn3ZGrQpLZHwM2198

User avatar
stevepusser
Posts: 12930
Joined: 2009-10-06 05:53
Has thanked: 41 times
Been thanked: 71 times

Re: HOWTO: Backport and host Debian & Ubuntu packages on the

#11 Post by stevepusser »

The extra architectures can be added, but they still seem to be not able to pull anything from the appropriate Debian repositories. I suspect we still have to send in an email to get those working, as well as a Stretch repository.

I'm not associated with Debian in any way; I'm just an enthusiast that started backporting my own packages, then got involved with packaging for the MEPIS and then MX Linux repositories. I usually pick up any backporting tricks & tweaks by studying how other people have accomplished the same thing. Failing that, it's up to experimentation. My avatar reflects how I feels when one of those experiments finally cracks a tough backport.
MX Linux packager and developer

Post Reply