Qubes Linux Utils

notes on the qubes-linux-utils repository with an emphasis on how to package or translate it into Guix

TODO:

Kernels

For building the kernel itself, see the qubes-linux-kernel repository. It has patches like qubes-vmm-xen, as well as the QubesOS kernel configuration.

This has utilities to package the kernel/build support files. The starting point for understanding this is kernel-modules/qubes-prepare-vm-kernel. The files that dom0 expects to have are described in "Managing VM Kernels". My kernel directories also have an undocumented empty file named memory-hotplug-supported. Support for that feature is available in the Guix kernel, based on the configurations in gnu/packages/aux-files/linux-libre, although it is not always activated. Under the assumption that this is basically just a flag indicating that the feature should be used, the Guix package should create it (because the QubesOS kernel configuration will be used to build the guest kernel).

An old note says that the doc page on "Template Instantiation" is relevant. It is not clear why I came to that conclusion.

Current plan is to build a store entry with the file layout that dom0 expects, move that into dom0, and set that as the kernel for the Guix guest. Hopefully it will "just work".

initramfs

Qubes uses dracut to build an initramfs. It does so by taking in shell scripts from a predefined directory. It doesn't look like Qubes is doing too much extra from a brief glance, but not using dracut might be expensive. Guix has an expression->initrd procedure in gnu/system/linux-initrd.scm which runs guile code instead.

dracut/full-modules

In normal AppVMs (at least PVH and HVM, for supported distros), /dev/xvdd contains some boot files (in particular, initramfs, vmlinuz, and a directory with modules). This script binds the modules directory in /dev/xvdd to the modules directory in the $NEWROOT filesystem - presumably, the root that will be seen when boot is completed as opposed to the early init boot. It's not clear that this is technically necessary for booting PVH machines. Adding a vmlinuz and initramfs to a new directory successfully loaded the kernel. It errored out to the "bournish" REPL because the initramfs was very minimal and not expected to work - but it did successfully load the "bournish" REPL without any logic to use /dev/xvdd, so what I provided "worked" as expected given the circumstances.

The "Managing VM Kernels" page has some more details about the files in /dev/xvdd. It looks like only vmlinuz is mandatory, which is probably good. (NOTE: The documentation states this, but when I created a new directory with only a kernel there was an error that initramfs was missing. I'm not sure if the initramfs needs to be provided through another mechanism, "not mandatory" is different than "removable with no other action". Not worth looking into right now since I want to build an initramfs in Guix anyway.) Guix looks up modules in the store (through /sys/current-system/kernel/lib/modules/$ver). This is done at runtime with an environment variable, see %modprobe-wrapper in the (gnu services) module.

module-setup.sh just installs mount_modules.sh.

dracut/full-dmroot && dracut/simple

It looks like these are 2 different implementations of the same initialization logic (as in, the init program that the kernel launches). The most recent change to full-dmroot/qubes_cow_setup.sh was also applied to simple/init.sh, so I'm assuming they are both maintained at the same priority even though it is not clear why both of them exist (for community-supported distros?). The full-dmroot version seems generally fancier (more descriptive logging functions, for example), which seems to primarily come from library functions provided by dracut. But the simple script should still work.

The simple init script has a number of responsibilities which can be mapped onto a Guix initramfs. Some of them can be mapped onto functionality that has priority support in Guix initramfs creation, such as mounting secondary file systems, and some of them need to go into the arbitrary g-expression pre-mount, such as enabling memory scrubbing.

Relevant Guix files:

grub

Configures grub to disable xen's page scrubbing feature. This is re-enabled after initial boot in dracut/xen-balloon-scrub-pages/scrub_pages.sh. Also in the simple init script.

kernel-modules

Despite its name, does not implement any kernel modules. One c file which fiddles with an ext filesystem. Lots of references to labels, but it's modifying inode entries so it's not just relabeling the entire filesystem.

Possibly related to creating modules.img?

imgconverter

"Toolkit for generating icons and images for Qubes OS."

Presumably*, this means graphical images not disk images. This probably doesn't need any special treatment.

* a quick glance at the code seems to support this presumption

─── gnu
    └── store
        └── 66qalq2h24ax12vp059fdjjahcmqp1pz-python-3.10.7
            └── lib
                └── python3.10
                    └── site-packages
                        ├── qubesimgconverter
                        │   ├── imggen.py
                        │   ├── __init__.py
                        │   ├── __pycache__
                        │   │   ├── imggen.cpython-310.opt-1.pyc
                        │   │   ├── imggen.cpython-310.pyc
                        │   │   ├── __init__.cpython-310.opt-1.pyc
                        │   │   ├── __init__.cpython-310.pyc
                        │   │   ├── test.cpython-310.opt-1.pyc
                        │   │   ├── test.cpython-310.pyc
                        │   │   ├── test_integ.cpython-310.opt-1.pyc
                        │   │   └── test_integ.cpython-310.pyc
                        │   ├── test_integ.py
                        │   └── test.py
                        └── qubesimgconverter-4.2.13-py3.10.egg-info
                            ├── dependency_links.txt
                            ├── entry_points.txt
                            ├── PKG-INFO
                            ├── SOURCES.txt
                            └── top_level.txt

not-script

Processes device additions/removals. Talks to xenstore, mounts loop devices (see man loop). Also see docs/misc/block-scripts.txt in the Xen source.

I assume it is named "qubes-block" because this is the "type" of device that's being attached, which I assume the QubesOS developers created as a pseudo-type to interface with QubesOS infrastructure properly and not cause conflicts with other things. Lots of assumptions there.

This directory is presumably named "not-script" because normally this directory would contain a shell script or similar, but it turns out that the only technical requirement is that the file is executable, so a binary works just as well.

─── etc
    ├── ld.so.cache
    └── xen
        └── scripts
            └── qubes-block

qmemman

Implementation Notes:

Installation Notes:

┌── sbin
│   └── meminfo-writer
└── usr
    └── lib
        └── systemd
            └── system
                ├── qubes-meminfo-writer-dom0.service
                └── qubes-meminfo-writer.service

qrexec-lib

Implementation Notes:

Installation Notes:
Just libqubes-rpc-filecopy and libqubes-pure. IIUC pure is a support library for rpc-filecopy, is it also used elsewhere?

┌── include
│   ├── libqubes-rpc-filecopy.h
│   └── qubes
│       └── pure.h
└── lib64
    ├── libqubes-pure.so -> libqubes-pure.so.0
    ├── libqubes-pure.so.0
    ├── libqubes-rpc-filecopy.so -> libqubes-rpc-filecopy.so.2
    └── libqubes-rpc-filecopy.so.2

udev

Guix provides a file->udev-rule procedure in (gnu services base). Udev rules are a service. Unclear if additional configuration will be needed.

Files other than udev rules are provided:

─── lib
    ├── qubes
    │   ├── udev-block-add-change
    │   ├── udev-block-remove
    │   ├── udev-usb-add-change
    │   └── udev-usb-remove
    ├── tmpfiles.d
    │   └── xen-devices-qubes.conf
    └── udev
        └── rules.d
            ├── 90-qubes-dmroot.rules
            ├── 99-qubes-block.rules
            ├── 99-qubes-misc.rules
            └── 99-qubes-usb.rules

Makefile Targets Table

┌──────────────────┬────────────────────────┐
│   imgconverter   │                        │
│    not-script    │     default targets    │
│     qmemman      │                        │
│    qrexec-lib    │                        │
├──────────────────┼────────────────────────┤
│       udev       │     default install    │
├──────────────────┼────────────────────────┤
│      dracut      │ fedora & debian kernel │
│       grub       │                        │
├──────────────────┼────────────────────────┤
│ initramfs-tools  │     debian kernel      │
├──────────────────┼────────────────────────┤
│  kernel-modules  │     fedora kernel      │
└──────────────────┴────────────────────────┘

Download the markdown source and signature.