Building Stuff From Source

by @bwasti


Probably the most valuable thing I've learned as a programmer is how to build things from source without getting (too) frustrated. *cough*

Building from source opens the door to bleeding edge technology that puts you well ahead of the discovery/adoption curve. You'll gain access to all the newest bugs and sometimes new features.

Here's a step-by-step playbook of my non-best practices:

Grab the code

Internet is fast nowadays, just --recursive and don't even think about it. Accidentally missing code tends to throw nonsense errors, so I just pull all the code every time.

git clone --recursive https://github.com/MAANG/someproj.git
cd someproj/

At this point many smart folks checkout a stable branch. I don't.

Breadcrumbs? (you see something like INSTALL.md)

The first thing I do is check if theres an INSTALL.md or INSTALLATION.md file.

Why? To check if there are any flags that will make my life easier. That's it.

For some reason, developers think other developers read installation instructions, but this is not the case. I try to avoid reading anything beyond build flags.

The default flags are sometimes crap. If you end up with a slow debug binary and every test in existence, you probably forgot to check the INSTALL.md.

cmake (you see CMakeLists.txt)

Easily the most popular build system, just pray you don't have to actually read or edit (let alone maintain) a cmake build. It's loaded with indecipherable foot-guns and compatability issues. Strings are lists are variable names. It's wild, but you can google most things.

Anyway, when you see CMakeLists.txt,

mkdir build
cd build
cmake .. # all the extra flags can go here

configure (you see a file called configure and it's green)

./configure and hope you don't hit errors. As with cmake, you're gonna want to check for INSTALL.md.

I have no idea how to actually fix broken configure files (something about autoconf? or some other anachronistic GNU pacakaging), but it's never proven to be much of an issue.

Pure make (you only see Makefile)

This is suspicious. Skip to the next bit about using make.

Bazel (you see .bzl files)

Prepare to install Java (in [current year]!) as well as the rest of the world. I tend to look for pre-builts binaries or alternative projects at this point.

Errors when running cmake .. or ./configure

Cool, a bunch of stuff just printed out and it looks like there are errors.

These errors are usually related to 1. your compiler being ancient or 2. required libraries not being available/found.

For 1, just update. For 2, you may need to build the library from source. See here for a quick guide.

Nothing got built when running cmake .. or ./configure

Nope, cmake just generates Makefile files (unless you specify -GNinja). Same with ./configure.

Some guides might recommend you build with cmake --build or the like. I never do that. I do this:

make -j$(nproc)

It's cooler and pretty much always works. Also, that -j is REALLY important, it parallelizes the build by the number passed in. cmake --build accepts -j too.

But what if you don't want to guess the optimal level of parallelism? Try to use ninja! Cmake can conveniently generate different types of build files.

pip install ninja
cmake .. -GNinja # all those flags you found in install.md
ninja

Now we're cooking with fire.

I haven't done the benchmarking to determine if its actually faster than make -j$(nproc) but it's usually faster than make wait 10 min Ctrl-C sigh make -j$(nproc).

Now it's built, but I can't run anything

I'm personally opposed to system installations.

There's no need to be like me, but I find it much more convenient to build something, play with it, and then blast it from existence with a simple rm -rf [folder]. Most things (especially bleeding edge software) break or change substantially over time, so removal is a priority for me.

So then how do I run anything? Typically after the build you end up with binaries in the bin/ folder and libraries in the lib/ folder.

export PATH="${PATH}:$(pwd)/bin"
export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:$(pwd)/lib"

And bang! you've got it made. Unless you need the include directories as well (set CPATH or something, I never use it).

If I find I'm setting these paths a lot, I tend to add them directly to my ~/.bashrc. It's kinda like I installed them into usr/lib or lib64 or share/lib or whatever the current popular spot is.

Frustration

To be completely honest, this part never goes away completely.

Better Practices

Folks over at hackernews have a lot of good comments, including mention of meson and autoconf installation flags. They can be found here: https://news.ycombinator.com/item?id=29125380