Crystal language... How does Debian compile it?

[tycode]

I'm trying to figure out how to bootstrap the Crystal language.

There's a bootstrap script on the upstream org, but it's unmaintained and after a few hours of fiddling and solving some basic problems, I'm not sure I'm going to get anywhere with this approach. (EDIT: I've since posted my findings trying to use the crystal bootstrap-script over on this upstream thread, where someone else attempted something similar a year ago).

Now I notice Debian bookworm (which I use as my main OS) has crystal 1.6.0 in apt. I have done some vague reading over the years on Debian's principles, how Debian packages things etc., though so far I've not had the experience of packaging something myself. But I get the impression that Debian only distributes software which they can compile from source with no binary blobs required, so this implies Debian has found a way to bootstrap Crystal already.

I looked up the Debian source for crystal, but it seems the only added files are in the "debian" directory, and the only one of interest seems to be the "rules" file. That is a very small file though and it's not really shedding any light for me on how this manages to build Crystal.

Can anyone give me some pointers? Am I right that somewhere in the world there is a machine that is, or has been, somehow bootstrapping and compiling Crystal purely from source, and pushing out crystal packages to apt, and if so how can I replicate this? Or am I missing something fundamental here?

Thanks in advance to anyone who can help.

[stevepusser]

The changelog says it's building it with LLVM/clang in Bookworm.

Code: Select all

crystal (1.6.0+dfsg-1) unstable; urgency=medium

  * New upstream version.
  * d/patches:
    - Refresh patch 'remove-check-in-random-failing-test'.
    - Add patch 'fix-sbuild-pkgbuilddir-filtering-in-unix-spec'
      (Closes: #1017565).
  * Switch to llvm-toolchain-14 (Closes: #1017662).

Slightly related, apparently thunderbird-esr 128 builds in Bookworm for i386 fail when using the default llvm-14 toolchain, and are succeeding in Bullseye using llvm-16. Probably someone needs to file a bug report. I'm gonna try and build an i386 package for MX repo using llvm-16 in the meantime.

[Aki]

Hello,

The upstream source code is here:

• https://github.com/crystal-lang/crystal

The source code available in Debian is here:

• sources / crystal / 1.12.1%2Bdfsg-1 (Debian Unstable)

• sources / crystal / 1.6.0%2Bdfsg-3 (Debian Bookworm)

Hope this helps.

[fabien]

You might also find some clues in the Build log for crystal.

[stevepusser]

I think the OP's question is that they thought that Crystal had to be compiled with the Crystal compiler, which is not the case. Debian uses the llvm compiler toolchain.

llvm is in turn compiled with gcc.

Gcc is compiled with a previous version of gcc.

Somewhere there's a gcc bootstrap, but it had to be a long time ago.

[donald]

Very interesting.

@Best_Threads

[blackbird]

Is it sure LLVM can compile Crystal code? Maybe the change log just says that the LLVM backend from Crystal is updated to a new version. Looking at the build log that fabian provided, it seems like Debian compiles Crystal with an existing Crystal compiler from the Debian repositories.

The Travis config could also provide some hints - https://salsa.debian.org/deiv/crystal/- ... type=heads.
There the default compiler is a blob from Github:

Code: Select all

    previous_crystal_base_url:
    description: "Prefix for URLs to Crystal bootstrap compiler"
    type: string
    default: "https://github.com/crystal-lang/crystal/releases/download/1.11.2/crystal-1.11.2-1"

[tycode]

Thanks for all the replies, everyone. I haven't yet had a chance to have another deep dive into this but this is already really helpful.

Is it sure LLVM can compile Crystal code? Maybe the change log just says that the LLVM backend from Crystal is updated to a new version. Looking at the build log that fabian provided, it seems like Debian compiles Crystal with an existing Crystal compiler from the Debian repositories.

This is what I was thinking, too. Particularly as this blog post from the authors of Crystal indicates that Crystal has required itself to build since 2013.

I took a quick glance at that log, and it does show it's installing crystal @ 1.6.0+dfsg-2 and building crystal @ 1.6.0+dfsg-3. It doesn't explain how crystal @ 1.6.0+dfsg-2 was built. Are build logs kept for all eternity? If they are, I might be able to reconstruct the history of how this was done by looking at the build logs of every version since when it was originally added to Debian.

One of my experiments yesterday was to start with just crystal @ 1.6.0 from apt, then clone the upstream repo ( https://github.com/crystal-lang/crystal/ ), check out various different tags (starting with 1.6.0 itself, then the latest patch of that and each minor since) and run make on each one. But this seems to do some black magic, as it leaves 2 files inside the repo dir ( .build/crystal and src/llvm/ext/llvm_ext.o ) and also affects the contents of $HOME/.cache/crystal, and the existence and contents of these files seem to affect future builds: when I deleted them all between each make, then it would only build up to a certain version and give errors on future versions (I'd have to re-test to find out which, it was all a bit confusing at the time), whereas when I left the files there and only ran git checkout and make commands, it was quite happy to keep building newer and newer versions, up to wherever I stopped, which was around 1.11 I think. This suggested to me that the build process is figuring out by itself where to find the previous just-built version, but giving me no info on what files affect this.

[blackbird]

This suggested to me that the build process is figuring out by itself where to find the previous just-built version, but giving me no info on what files affect this.

I think it takes the last version to be used for compilation from here: https://github.com/crystal-lang/crystal ... /shell.nix.
For Debian, the intuitive way would be also to use the latest former version of Crystal to compile the next, just using the Debian package instead of the blob from Github.

[stevepusser]

Interesting:

Wikipedia states:

The Crystal compiler was first written in Ruby, but later rewritten in Crystal, thus becoming self-hosting, as of November 2013.[10]

and

Although resembling the Ruby language in syntax, Crystal compiles to much more efficient native code using an LLVM backend, at the cost of precluding the dynamic aspects of Ruby.

[stevepusser]

If you go back to the very first 0.34.0 version to appear in Debian, you would ask how did they do it without already having crystal packages available.

They just used llvm. Here are what the build-depends were:

Code: Select all

Build-Depends: debhelper-compat (= 12),
               git,
               libbsd-dev,
               libedit-dev,
               libevent-dev,
               libgc-dev (>= 1:7.6.0),
               libgc1c2,
               libgmp-dev,
               libgmpxx4ldbl,
               libllvm9 | libllvm8,
               libpcre3-dev,
               libssl-dev,
               libtool,
               libxml2-dev,
               libyaml-dev,
               libz-dev,
               lld-9 | lld-8,
               llvm-9-dev | llvm-8-dev,
               pkg-config

This is when it depended on itself:

Code: Select all

crystal (1.2.1+dfsg-2) unstable; urgency=medium

  * d/control:
    - Update standars version (no changes needed).
    - Point Vcs-* fields to the updated git repository.
    - Update maintainer to debian account.
    - Update LLVM version.
    - Rename packages to crystal (remove -lang).
    - Add build depend on itself.
  * Update d/copyright.

 -- David Suárez <deiv@debian.org>  Sat, 30 Oct 2021 15:03:25 +0200

[tycode]

Interesting.

I downloaded the sources from here: https://snapshot.debian.org/package/cry ... %2Bdfsg-1/

and again, didn't find anything that would successfully bootstrap this.

debian/changelog says:

Code: Select all

crystal (0.34.0+dfsg-1) unstable; urgency=medium

  * Initial release (Closes: #822275).

 -- David Suárez <david.sephirot@gmail.com>  Fri, 10 Apr 2020 17:26:40 +0200

So I looked up bug822275, and there is this commentary:

From: Luc Sarzyniec <debian@olbat.net>
To: submit@bugs.debian.org
Subject: RFP: crystal -- Compiler of the Crystal object-oriented programming language
Date: Fri, 22 Apr 2016 23:46:18 +0200

Package: wnpp
Severity: wishlist

* Package name : crystal
Version : 0.15.0
Upstream Author : Manas Technology Solutions <info@manas.com.ar>
* URL : http://crystal-lang.org/
* License : Apache-2.0
Programming Lang: Crystal
Description : Compiler of the Crystal object-oriented programming language


The Crystal's language syntax is inspired by Ruby, the language is statically
type-checked but does not require that the type of variables or method
arguments be specified.

The project's sources are hosted on GitHub [0].

I started to work on the Debian packaging of the project [1], but I still have
some issues to address. The first one: the compiler is written in Crystal so
crystal has to be specified in the package's build dependencies - I don't
really know how to handle it. This work may be used as a base to build
something cleaner.

[1] just links to https://github.com/olbat/crystal which doesn't seem to have any additional commits beyond upstream.

Looking in the source dir of this 0.34.0 version, there is only one C/C++ file, that is the llvm_ext.cc, which doesn't contain very much at all, everything else is Crystal.

Again debian/rules doesn't seem to shed any light on how this is compiled so I assume it just is just running make inside the source dir.

Doing so gives me an error that it can't find llvm-config. Running LLVM_CONFIG=/usr/bin/llvm-config-14 make instead gives me a bunch of errors compiling llvm_ext.cc, indicating I'd probably need an older LLVM.

So I just quickly hacked the Makefile to prevent it trying to build llvm_ext.cc to see what would happen.

It invoked bin/crystal, which is a shell script that tries to find crystal, which found my system-installed crystal-1.6.0.

After doing apt remove crystal, trying the make again results in the message "You need to have a crystal executable in your path!"

So it would seem to me that in order to build this 0.34.0 version, they had to do some bootstrapping manually (perhaps following the bootstrap-script I originally mentioned, which no longer works), and maybe the details of how this was done have been lost.

I might try installing some old Debian versions in VMs, and see if the bootstrap-script works anywhere there.

[Popish]

You’re correct that Debian must be using a pre-built Crystal (or some other version of Crystal) to bootstrap the compiler, but you’ll need to dive into the specific build steps by looking at the debian/rules file and understanding the dependencies involved. If you want to replicate the bootstrapping process, focusing on the build logs, experimenting with building an older version of Crystal, and using that to bootstrap the newer version will be key.