Building and Running Linux on Rutgers iLabs
Since I had some free time on my hands I decided to do something I’ve wanted to do for a while and mess around with the Linux kernel. With my time at Rutgers slowly running out, I’ve wanted to do this on their iLabs. These are computers that CS students can use or SSH into and some of them are a lot beefier than I’d buy on my own.
This post will be a guide to some of the basic things I did while working on the iLabs. Since there’s no shortage of Getting Started posts about building the kernel, I’ll focus this one specifically on getting things running on the Rutgers iLabs in the simplest way possible.
Some initial considerations
I’m going to assume you have basic knowledge of Linux and the shell before starting. But before we do that, we need to make a couple of decisions.
First, what computer you’ll use. Rutgers has a good amount of iLabs available of various specs with online monitoring so you can check what the usage is before joining. In general, you can just grab the one with the largest number of cores that’s free.
The second is where you want to store things. You have a number of
options but most of those won’t be viable since you should plan to use
at least 15GB. The best option, what I used, is probably /freespace/local
which is machine-specific and not backed up, but doesn’t generally have
restrictions.
Next is how many cores to use when you build. You can find out the number of
vCores with nproc but if there’s other people using the machine it might not
be wise to use up all its resources. Whenever you run make you can instead run
make -j N where N is the max processes to split across.
I did all my work over SSH, so you shouldn’t need an X environment or have to use X2Go.
And last keep in mind things might have changed since I posted this, so some of these instructions might be a little outdated.
The initial build
You’ll need to download a copy of the linux kernel kernel. You can grab the latest stable source from kernel.org (or from a the git repos or a mirror). The actual version shouldn’t matter much but make sure it’s not too old or the instructions might change. I used the 5.5.13 tarball.
Since the iLabs already have all the necessary build tools, the actual build is pretty straightforward.
From within that directory run make distclean which will move any excess files
that happen to be left over. We need to configure the kernel and for our
purposes the default configuration is fine so run make defconfig. Then just
run make (with the -j option as mentioned before) and grab some coffee while
everything builds. To rebuild you just run make again.
The end result is a file arch/x86_64/boot/bzImage. That’s the self-extracting
kernel image with all the kernel code. Note the default configuration will
build Linux for the same architecture as the machine it’s being built on, which
should be x86_64 for the iLabs.
Running the emulator
The iLabs have QEMU which is an emulator with special support for Linux which makes emulation a bit easier to setup. We can run the kernel as follows.
$ qemu-system-x86_64 -kernel arch/x86_64/boot/bzImage -nographic -append "console=ttyS0"
Here’s what each part of the command means,
qemu-system-x86_64is the command to start QEMU. The iLabs have emulators installed for a bunch of architectures but we’re trying to avoid doing cross compilation, so we use x86_64.-kernel arch/x86_64/boot/bzImagetells QEMU to load the kernel image after it starts. Normally Linux is loaded by a bootloader which sets up the environment but QEMU has special support which makes things easier.-nographicprevents a video display from being attached. Since I’m doing this over SSH I passed this to prevent the need for a X session, but if you’re in an X session this is unnecessary.-append "console=ttyS0"works with-kernelto also give Linux command line options, in this case to output the console to the serial port. This again is only necessary because I don’t have an X session, but the end result is you can interact with the console via the stdin and stdout of QEMU.
Here’s what happens when we run it.
[ 5.652630] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 5.653698] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.5.13 #1
[ 5.654288] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 0.5.1 01/01/2011
[ 5.655164] Call Trace:
[ 5.656417] dump_stack+0x50/0x6b
[ 5.656812] panic+0xf7/0x2c6
[ 5.657165] mount_block_root+0x173/0x22c
[ 5.657704] mount_root+0x10e/0x12c
[ 5.658083] prepare_namespace+0x138/0x167
[ 5.658527] kernel_init_freeable+0x1c4/0x1e6
[ 5.658971] ? trace_event_define_fields_initcall_finish+0x62/0x62
[ 5.659576] ? rest_init+0xa0/0xa0
[ 5.659908] kernel_init+0x5/0x100
[ 5.660250] ret_from_fork+0x22/0x40
[ 5.661198] Kernel Offset: 0x17a00000 from 0xffffffff81000000 (relocation range: 0xffffffff80000000-0xffffffffbfffffff)
[ 5.662507] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0) ]---
The kernel will start, but will panic, means it enters a state it doesn’t know how to recover from. Before I fix what’s going on I want to take you on an aside to the boot process.
The boot process
When we boot up physical hardware the first code our CPU will run will be setup code stored on a ROM probably somewhere on the motherboard. This is hardware-specific code that sets up hardware, memory and whatever else has to be done then searches for a bootable device which it loads in and passes control to.
The code loaded from that device is generally a bootloader which then has the job of actually starting the operating system by loading the kernel. It makes sense to keep this portion separate from the kernel since a lot of configuration needs to happen before kernel code can even run. For instance, our kernel may be built for x86_64 but actual x86_64 CPUs start in real mode which is an outdated 16-bit compatibility mode.
After the kernel runs and does its own setup and configuration it also needs to know what to do next. What the kernel wants to do, as shown by the error, is mount a root filesystem. After which it will run an init process from that filesystem which gets the responsibility of bootstrapping the rest of the system.
We need to pass the kernel something called an initial ramdisk or initramfs. This is an archive containing all the files the rest of the OS needs to bootstrap itself. When Linux is given an initramfs it will load it into a ramfs, essentially just a volatile filsystem stored in memory and look for the init process which it starts as root.
Creating an initramfs
You should create a new directory that will be the file structure of initramfs. Since we’re loading into a pretty limited environment, we’re limited in what we can run. And although we could put whatever we want in the initramfs, to keep things simple we just want to run small static binaries.
The obvious solution is busybox which is a collection of basic unix
utilities like sh, ls and vi. If you go to Download Binaries on their
website you should be able to find the statically linked busybox binary which
is the conglomeration of all the busybox tools.
After downloading it if you run ./busybox --list you get the complete list of
all tools. You can then copy busybox into a bin subdirectory in your initramfs
directory and install busybox so all its command are symlinked to the binary.
./busybox --install .
Busybox will check the command name it was run by and automatically act like that command.
Busybox comes with init which follows the System V standards. It requires a
bunch of configuration files in /etc to be setup to tell it what to do, but
we’ll go with a simpler solution of just initialing with a shell script.
We’ll then set up an init script called init at the root of your initramfs
that just opens busybox’s shell. Linux looks in a few places for the init
command, but /init comes before /bin/init so there’s no need to delete
busybox’s init.
#!/bin/sh
exec sh
The first line of the file tells the kernel to open this with /bin/sh, so the
actual init command becomes /bin/sh /init. The shell will then read this file,
ignore the first line, and following the second line will replace itself with a
new instantiation of the shell.
Before making the archive you need to ensure all these files are marked as executable.
The kernel only accepts cpio archives since that’s considered standard by POSIX, even though it’s a somewhat esoteric format nowadays. The command to create archives is equally esoteric requiring you to pass files in via stdin.
From within your initramfs directory the archive can be created with something like the following. The exact command depends on the directory structure you’re using.
$ find . | cpio -ov -H new > ../initramfs.cpio
Running the kernel with initramfs
We can let QEMU take care of passing Linux the initramfs, it just gets passed as another parameter. It’s worth noting an initrd, which is the name of the option, is an older and subtly differnt different way of bootstrapping the kernel. The kernel recognizes the cpio format and loads it as an initramfs.
qemu-system-x86_64 -kernel linux/arch/x86_64/boot/bzImage -initrd initramfs.cpio -nographic -append "console=ttyS0"
This should boot successfully and we shuould have access to all the busybox
commands you set up earlier. We have a very limited environment because
normally after boot we’d do a lot more to setup the system. For instance if we
want the /proc and /sys filesystems we have to mount them manually.
# mkdir proc sys dev
# mount -t proc none /proc
# mount -t sysfs none /sys
Just like the kernel panics if it can’t find initramfs, if our init process
exits we’ll get a similar error. The proper shutdown is poweroff -f which
tells the kernel to initiate its shutdown process.
Compiling Programs for our Linux
Let’s say we want to compile and run a program to print Hello World.
#include <stdio.h>
int main() {
puts("Hello, World!\n");
return 0;
}
We can build this with gcc, copy it into the initramfs and rebuild, then run it from inside the emulator.
# hello
sh: hello: not found
This is a weird error (the file exists and is in the right place) but what’s
happening is gcc dynamically links everything by default. And the files we’re
dynamically linking to aren’t there since hello is built for the host machine,
not the guest.
The solution is to build with -static.
$ gcc -static hello.c
/usr/bin/ld: cannot find -lc
collect2: error: ld returned 1 exit status
But this doesn’t work either since the iLabs don’t have the static version of
libc installed anywhere gcc checks. Some searching around reveals, however, they
do have it somewhere. Once you find where it is you can include it with -L.
$ locate libc.a
/opt/mpss/3.8.2/sysroots/k1om-mpss-linux/usr/lib64/libc.a
/usr/lib/debug/usr/lib64/libc.a
/usr/lib/x86_64-redhat-linux6E/lib64/libc.a
$ gcc -L/usr/lib/debug/usr/lib64 hello.c
$ ./a.out
Hello, World!
Since a.out is statically linked it works fine in the emulator.
Hello World in dmesg
Now that we have a basic environment to play around with we can start modifying and testing parts of the kernel. To start, let’s print Hello World at the beginning of kernel initialization.
To do this just add one line to the start of start_kernel in init/main.c.
pr_notice("Hello, World");
Rebuild and, if you put it at the top, it will be the first output from the kernel.
SeaBIOS (version 1.11.0-2.el7)
iPXE (http://ipxe.org) 00:03.0 C980 PCI2.10 PnP PMM+07F95610+07EF5610 C980
Booting from ROM...
[ 0.000000] Hello, World
[ 0.000000] Linux version 5.5.13
[ 0.000000] Command line: console=ttyS0
Changing something more substantial
Hello world is nice but we want to do something more significant. If we look for
a something to mess with in the
syscall list a getrandom appears as a good option.
If we to change something more fundamental like open or read we’d very
likely break something since all our busybox tools depends on their behaviour.
Since the standard way to get randomness in Linux prior to this syscall was
/dev/random and /dev/urandom, and it took a while for the
syscall to even appear in libc, it’s unlikely this is even being used anywhere.
Anyway, the entrypoint in the kernel for this syscall is in
drivers/char/random.c which we can find by just grepping around.
SYSCALL_DEFINE3(getrandom, char __user *, buf, size_t, count,
unsigned int, flags)
{
int ret;
if (flags & ~(GRND_NONBLOCK|GRND_RANDOM))
return -EINVAL;
if (count > INT_MAX)
count = INT_MAX;
if (!crng_ready()) {
if (flags & GRND_NONBLOCK)
return -EAGAIN;
ret = wait_for_random_bytes();
if (unlikely(ret))
return ret;
}
return urandom_read_nowarn(NULL, buf, count, NULL);
}
Our change will essentially just be another Hello World. Instead of getting random bytes, we just repeatedly fill the user’s buffer with Hello World. Technically we’re not really doing anything wrong since by nature of randomness this is expected behaviour, but this would obviously hurt security if we used our kernel for anything important.
SYSCALL_DEFINE3(getrandom, char __user *, buf, size_t, count,
unsigned int, flags)
{
int ret;
static char *hello = "Hello World!";
if (flags & ~(GRND_NONBLOCK|GRND_RANDOM))
return -EINVAL;
if (count > INT_MAX)
ret = INT_MAX;
count = INT_MAX;
} else {
ret = (int) count;
}
while (count > 0) {
size_t s = 12;
if (count < 12) s = count;
if (copy_to_user(buf, hello, s))
return -EFAULT;
c -= s;
buf += s;
}
return ret;
}
Instead of immediately building, we’ll create a test program and run it before and after.
#include <stdlib.h>
#include <stdio.h>
#include <sys/syscall.h>
int main(int argc, char **argv) {
void *buffer;
if (argc != 2) {
fprintf(stderr, "Usage: gerandom <bytes>\n");
exit(1);
}
size_t size = strtol(argv[1], NULL, 10);
buffer = malloc(size);
ssize_t r = syscall(SYS_getrandom, buffer, size, 0);
fprintf(stderr, "getrandom(%p, %lu, 0) = %ld\n", buffer, size, r);
fwrite(buffer, size, 1, stdout);
exit(0);
}
We have to use syscall since the getrandom library function wasn’t included in
glibc until 2.25. As of this post the iLabs only compile with 2.17. Be sure to
statically link this as shown before.
On both the iLabs and the unmodified kernel this should give random data of the number of bytes specified, but in the emulator with the modified kernel we’ll see something like this.
# getrandom 20 | xxd
00000000: 4865 6c6c 6f20 5765 726c 6421 4865 6c6c Hello World!Hell
00000010: 6f20 576f o Wo
Random Resources
I plan to keep hacking around for a bit and wanted to link to resources that have been helpful or that include things I’ve done but not included in this post.
One cool book I’ve been following is Linux from Scratch. It shows you how to, from scratch, build a complete Linux system including trying to follow the LSB standards so it’s actually a proper Linux system. What this post shows is a really quick and dirty way to test our kernels and many programs won’t work in them since they expect certain processes and filesystems to be setup correctly.
To remove some of he boilerplate of developing Linux modules there’s Linux Kernel Module Cheat which contains tons of tools for building Linux and emulating it in a simple environment. I couldn’t get this running in the ilabs, but I’d be curious if someone else is able to (shoot me an email).
I’ve also followed parts of Linux Device Drivers which is free on LWN. Creating device drivers is the most useful part of kernel development to learn since that’s where most the development happens, and where patches are most likely to get accepted (as opposed to the more core parts of the kernel). It’s also what most of the kernel code is.
And I’ve read parts of Linux Insides which is still in progress but has a lot of great well-written information about how parts of the kernel work. This probably isn’t as useful for kernel development as something like Linux Device Drivers but it satisfies an itch to learn how things work, and I’m just doing this for fun anyway.
I hope someone finds this at least a little fun or useful. I don’t keep comments on this blog but if you want to reach out to me, even if just to tell me you enjoyed my post, my contact info is in the about page linked in the header.