Bits of networks

Bits of networks

Networking, system, research

17 Aug 26

Patching OpenSSH in Debian

This article was written entirely without the help of any generative AI tool.

I needed to test an OpenSSH patch. I thought that rebuilding the Debian package would be easy, but it wasn't.

My case for rebuilding the OpenSSH Debian package

My main use-case when building or rebuild packages is to use patched or backported versions locally or for internal server use, not to push them in the Debian archive. It means I don't need the same kind of quality process, but I would still like the resulting packages to be as close to Debian as possible, hoping it helps my system to remain stable.

I love Debian for its stability and community, but there's one thing that has always irritated me: there are so many available tools to build Debian packages, and there is no particular recommendation to use any of them. Besides, the whole building process feels quite obscure and magical.

In that particular case, I wanted to test a patch to the OpenSSH client. Since I use SSH daily, I definitely don't want to run a weird build that could break any of my workflows. I thought I was starting to master the art of building Debian packages, but there were new interesting surprises!

Rebuilding Debian packages is easier than before

In recent years, the ease of building Debian packages has improved, notably because many packages are now maintained as a git repository.

Obtaining and modifying the source Debian package is easier for me thanks to git:

That last step can still be difficult, but at least it's nice to be able to work with a familiar git environment with branches, stashes, cherry-picks, etc.

In addition, there is (yet another) wrapper to facilitate working with Debian packages that are maintained in git repositories: git-buildpackage. It can integrate with pbuilder to build package in a chroot.

I found this particular guide to be quite useful: https://people.debian.org/~debalance/packaging-with-git.html. The Debian maintainer guide is a good resource too.

Hitting a wall at first

Armed with that knowledge, I quickly tried to build my patched OpenSSH package.

Let's clone the repository and apply our patch:

$ git clone https://salsa.debian.org/ssh-team/openssh.git
$ git checkout trixie
$ git apply my-openssh-patch.patch
$ git commit -am "Apply local patch to fix race condition"

Then we need to initialize a fresh pbuilder chroot (see the guides above):

$ DIST=trixie git-pbuilder create

Now let's try to build in the chroot:

$ DIST=trixie gbp buildpackage --git-ignore-branch
dpkg-source: info: building openssh using existing ./openssh_10.0p1.orig.tar.gz
dpkg-source: info: using patch list from debian/patches/series
dpkg-source: info: local changes detected, the modified files are:
 openssh/clientloop.h
 openssh/mux.c
 openssh/ssh.c
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/openssh_10.0p1-4.diff.R0ebST
dpkg-source: hint: make sure the version in debian/changelog matches the unpacked source tree
dpkg-source: hint: you can integrate the local changes with dpkg-source --commit
gbp:error: 'git-pbuilder' failed: it exited with 2

OK, it seems that we need to maintain the patch with quilt instead of committing the code change directly to the git repository. This is annoying but doable. I must admit I did it by hand by simply copying the patch and editing debian/patches/series, and committing the patch (replacing the previous commit).

Let's try again:

$ DIST=trixie gbp buildpackage --git-ignore-branch
dpkg-source: info: local changes detected, the modified files are:
 openssh/clientloop.h
 openssh/mux.c
 openssh/ssh.c
dpkg-source: error: aborting due to unexpected upstream changes, see /tmp/openssh_10.0p1-4.diff.R0ebST

In short, we get the same error, even though this time I didn't touch the actual openssh code in the git repository. I verified several times that I didn't make a mistake somewhere, but no, the only commit I added is touching debian/patches, not the upstream code directly.

At the point, I'm in WTF mode, and I leave this rest for a while.

Hunting for the culprit

Several days later, looking at the problem again, I found a blog post from Colin Watson, OpenSSH's Debian maintainer, precisely about converting the OpenSSH package to the "new" (that is, new as in 2010) quilt format: https://www.chiark.greenend.org.uk/~cjwatson/blog/thoughts-on-3.0-quilt-format.html

According to his article, Colin takes the unusual (but well-motivated) approach of having an already-patched tree in the main git branch, while other maintainers prefer having their git repository reflect the upstream code: they manage patches either with quilt or in a separate branch. Due to this choice, Colin says "it’s a bit awkward to set things up when checking out from revision control", because quilt doesn't understand that the patches are already applied. AH AH! It seems we got our culprit!

The article mentions an helpful script that sets things up properly, let's use it:

$ debian/rules quilt-setup
dh quilt-setup --with=runit 
dh: error: Unknown sequence quilt-setup (choose from: binary binary-arch binary-indep build build-arch build-indep clean install install-arch install-indep)
make: *** [debian/rules:122: quilt-setup] Error 255

Err, so the helpful script doesn't exist!

Looking around some more, I found out that the script was removed in 2014 with a commit message "Initialize git-dpm". We are on a good follow-up trail here.

Trying out git-dpm

So, let's learn to use (yet another) tool to manage Debian packages in a git repository!

The manpage for git-dpm is actually rather helpful. The only confusing aspect is that Colin is again storing the applied state of patches in the git tree, while git-dpm defaults to store the unapplied state (i.e. upstream state). It means the branches explanations from the manpage don't actually apply to the Debian openssh repository. It's a choice to make when initializing a new repository, see the --patches-applied option).

Anyway, with the right tool, it's now straightforward to patch and build the package:

$ git-dpm checkout-patched
$ git cherry-pick XXX  # or git am, git apply + commit, etc
$ git-dpm dch -- -i
$ DIST=trixie gbp buildpackage --git-ignore-branch

And I got my patched package openssh-client_10.0p1-7+deb13u4.2_amd64.deb!

Conclusion

Building Debian package still feels obscure, but at least I learned new tools to manage packages in git.

It's a bit annoying that different Debian maintainers have different habits, but I feel the ecosystem is already more homogenenous than it was a few years ago. Besides, once you understand the maintainer's logic (especially through their useful blog posts!), things become easier.

About git-dpm, there is a small downside for me: most commits are merges, which makes it difficult to review historical diff. My first reflex to understand the packaging of openssh was to look at the commit history:

$ git log -p debian/rules

but this was mostly useless: I could see the introduction of quilt-setup in 2010 but not its removal afterwards, WTF.

It turns out that git doesn't display diff for merges by default, but you can ask for them:

$ git log -p --first-parent debian/rules

That's already more useful, and one more thing I learnt during this small experience!