Intro

Since reading the excellent systemd has been a complete, utter, unmitigated success and discovering Lennart Poettering’s Pid Eins blog, I’ve been on a bit of a streak of playing around with developing services and managing them with systemd:

Today we’re looking at another component in the systemd ecosystem, namely Portable Services.

Despite being available since v239 (all the way back in 2018), the first I came across it was digging around Pid Eins after reading the aforementioned blog post.

So what is it?

The overview in the Portable Service docs provides a pretty clear definition:

systemd (since version 239) supports a concept of “Portable Services”. “Portable Services” are a delivery method for system services that uses two specific features of container management:

  • Applications are bundled. I.e. multiple services, their binaries and all their dependencies are packaged in an image, and are run directly from it.
  • Stricter default security policies, i.e. sand-boxing of applications. The primary tool for interacting with Portable Services is portablectl, and they are managed by the systemd-portabled service.

It’s worth reading the rest of the overview to understand the rest of the details.

As someone who works a lot with docker/podman images, this seems to cover a lot of the same usecases, although with some slightly different design decisions:

  • A simpler format (‘just’ an OS tree, either as a folder or as an image file)
  • Native integration with systemd
  • No requirement for a local or remote image store
  • Automatic handling of multiple services from one image
  • Services are run as regular processes instead of PID 1

One way to summarise this would be ‘docker but less magic’ - systemd services can handle most common docker usecases (isolated filesystems, network namespaces, SETUID…) but for the most part those options are opt-in rather than defaults.

There’s areas where each has its strengths. For example, docker sets up network bridges for you, while systemd requires a lot more configuration - meanwhile systemd makes it easier to override settings via drop-in units without modifying the original definition, something that docker doesn’t natively have a provision for.

Of course, docker is much more prevalent than portable services, so you’re more likely to find guidance and images tailored to that. A bunch of platforms (managed k8s for example) will support running docker images but not Portable Services.

Which set of tradeoffs you go for will come down to a bunch of factors. Regardless, it’s useful to understand different technologies - so let’s dive in!

Involving Nix

Note: this article assumes some level of familiarity with Nix and nixpkgs - If you’re not already familiar with both, I’d recommend starting from NixOS , or following the excellent guide from fasterthamline for a deeper dive.

Nix is great at certain things: dependency tracking and hermetic builds being some of them.

Deploying nix-based builds to nix-based machines is also great - however, because nix relies on the /nix/store approach to storing its artifacts, it’s a lot more painful if your server isn’t already running nix in some form.

In that case, you’ve got a few options:

  • Install nix on the server, with all the associated changes to the server (users, groups, builder services…) and the associated security considerations
    • Make sure you come up with a strategy for regularly garbage-cleaning the store!
  • Use something like nix-portable to bundle nix-based apps
    • An approach that uses a self-extracting archive on the target machine, which might be slow for larger tools
  • Run nix without installing it, which will set up an independent nix/store folder and chroot into it
    • This can be a pain, as you have to make sure that chroot is active whenever you run a binary
    • This article by Zameer Manji goes into more detail on how to do this

Using Portable Services to deliver a nix-based build is a compelling proposition: the build machine can run the build, work out the complete dependency set, and copy those files from its /nix/store into a squashfs filesystem image along with the necessary service files.

The target machine can then take that image and run it directly via a portablectl, without needing to have nix installed!

As it turns out, nixpkgs has a builder that does exactly that: pkgs.portableService.

pkgs.portableService takes a name, version number and a set of service files, and builds an image containing:

  • The service files
  • A /nix/store with the binaries and their their transitive dependencies
  • And the various other files required by the Portable Service spec.

The project

To give this a proper test, let’s try to set up something simple but non-trivial:

  • A modified version of caddy with additional plugins (built with nix)
  • In a self-contained portable service file
  • Run in an isolated environment (chroot / network namespace)
  • But still able to bind to privileged ports (via socket activation)

If you want to follow along with the code, you can find the example code on GitHub at JaimeValdemoros/portable-service-demo. Each commit corresponds roughly to one change in the article.

Let’s dig in!

Prerequisites

First things first, we need to be sure we can actually run portable services on the server we’re planning to deploy to.

Portable services are managed using the portablectl command, so the easiest way to check is to see if that command is installed:

which portablectl
# /usr/bin/portablectl

The debian system I was testing on had systemd but not portablectl, so I had to install the systemd-container package to make it available:

 apt install systemd-container

Setting up the project

Let’s get started with a flake-based build, with hello as our starting entrypoint. We’re not creating a portable service yet, just testing with the tool we’re going to wrap up.

# flake.nix
{
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
    };
    outputs = { self, nixpkgs, flake-utils }:
        flake-utils.lib.eachDefaultSystem (system:
        let
            pkgs = nixpkgs.legacyPackages.${system};
        in {
            defaultPackage = pkgs.hello;
        }
     );
}

nix run
# Hello, world!

Good start! Let’s wrap it in a portable service builder, while also pulling squashfs into our dev environment we can inspect the output

# flake.nix
{
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
    };
    outputs = { self, nixpkgs, flake-utils }:
        flake-utils.lib.eachDefaultSystem (system:
        let
          pkgs = nixpkgs.legacyPackages.${system};
          hello-service = pkgs.writeText "hello.service" ''
              [Unit]
              Description=Hello world

              [Service]
              Type=oneshot
              ExecStart=${pkgs.hello}/bin/hello

              [Install]
              WantedBy=multi-user.target default.target
          '';
        in {
          defaultPackage = pkgs.portableService {
              pname = "hello";
              version = "1.0";
              units = [ hello-service ];
          };
          devShell = with pkgs; mkShell {
              buildInputs = [squashfsTools];
          };
        }
     );
}

nix build
nix develop --command unsquashfs result/*.raw
# Parallel unsquashfs: Using 22 processors
# 1013 inodes (1003 blocks) to write
# [=================================================================================================================================|] 2016/2016 100%
# created 995 files
# created 281 directories
# created 18 symlinks
# created 0 devices
# created 0 fifos
# created 0 sockets
# created 0 hardlinks
tree -L 4 squashfs-root/
# squashfs-root/
# ├── dev
# ├── etc
# │     ├── machine-id
# │     ├── os-release
# │     ├── resolv.conf
# │     └── systemd
# │            └── system
# │                 └── hello.service
# ├── nix
# │     └── store
# │            ├── 30n8yzgji9ph8i5hshla57zbvdcpl5n3-libidn2-2.3.8
# │            │     └── ...
# │            ├── 8p33is69mjdw3bi1wmi8v2zpsxir8nwd-glibc-2.40-66
# │            │     └── ...
# │            ├── 90z6lxmraj3i1mvnzkdjwc1gdqxisbf4-libunistring-1.3
# │            │     └── ...
# │            ├── f6famnry0piih1d3hh6y73fxx05jxsa8-xgcc-14.3.0-libgcc
# │            │     └── ...
# │            ├── rvm13q97dv002vpkagsp62wfz9glrd5c-root-fs-scaffold-1.0
# │            │     └── ...
# │            └── s9p0adfpzarzfa5kcnqhwargfwiq8qmj-hello-2.12.2
# │                 └── ...
# ├── proc
# ├── run
# ├── sys
# ├── tmp
# └── var
#      └── ...
cat squashfs-root/etc/systemd/system/hello.service
# [Unit]
# Description=Hello world
# 
# [Service]
# Type=oneshot
# ExecStart=/nix/store/s9p0adfpzarzfa5kcnqhwargfwiq8qmj-hello-2.12.2/bin/hello
# 
# [Install]
# WantedBy=multi-user.target default.target

This has created pretty much what we would expect if we’re already familiar with nix:

  • A /nix/store folder with the various dependencies copied in
  • A series of empty folders described in the systemd docs
  • A service file at /etc/systemd/system/hello.service pointing to the binary (in /nix/store) to run

Let’s give it a try:

portablectl attach "$(readlink ./result)/hello_1.0.raw"
# (Matching unit files with prefix 'hello'.)
# Created directory /etc/systemd/system.attached.
# Created directory /etc/systemd/system.attached/hello.service.d.
# Written /etc/systemd/system.attached/hello.service.d/20-portable.conf.
# Created symlink /etc/systemd/system.attached/hello.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
# Copied /etc/systemd/system.attached/hello.service.
# Created symlink /etc/portables/hello_1.0.raw → /home/watchie/.local/share/nix/root/nix/store/8yx3zvsvg8rvc18k7fcy9yjj35sr75hy-hello-img-1.0/hello_1.0.raw.
systemctl start hello
journalctl -u hello
# Sep 06 21:22:38 smallweb systemd[1]: Starting hello.service - Hello world...
# Sep 06 21:22:38 smallweb hello[2323113]: Hello, world!
# Sep 06 21:22:38 smallweb systemd[1]: hello.service: Deactivated successfully.
# Sep 06 21:22:38 smallweb systemd[1]: Finished hello.service - Hello world.

Nice!

So what have we done here? We’ve:

  • Created a .raw file with a squashfs containing hello and all its dependencies
    • By only writing a service file, and letting nix handle working out what else needs to be in the image
  • Installed it on a host system
    • that doesn’t necessarily have nix installed
  • Started the service and seen its output in the host system’s journald logs

You might be wondering why we’re calling readlink before passing the path to portablectl attach - can’t we just attach to ./result/hello_1.0.raw?

The systemd docs say:

Note that the images need to stay around (and in the same location) as long as the portable service is attached. If an image is moved, the RootImage= line written to the unit drop-in would point to an non-existent path, and break access to the image.

The result symlink changes every time we run nix build, but the output folder in /nix/store will stay the same - at least until we run a garbage collection (unless we store it as a GC root). So it’s preferable for portablectl to symlink to the full path in /nix/store, rather than via the transient symlink in result/.

Factoring out the systemd file

While nix lets you write file contents inline, I find it’s more pleasant to have langauge-specific content in separate files, particularly so my IDE can apply language-specific highlighting.

If we pull out the service definition into a separate file, we can’t reference ${pkgs.hello} directly, so we need another way to point to the binary path. We’ve got two main options:

  • Add some kind of template directive in our service file, and use nix to template in the full path, as suggested in this gist by Github user gdamjan
  • Place the binary in a known path in the output image, as suggested in the pkgs.portableService docs, then point the service file at that path.

Let’s look at each in turn:

Known output path

The symlinks option in pkgs.portableService lets us put binaries in a well-known location like /bin/hello:

# within flake.nix
defaultPackage = pkgs.portableService {
	pname = "hello";
    version = "1.0";
    units = [ hello-service ];
	symlinks = [
		{ object = "${pkgs.hello}/bin/hello"; symlink = "/bin/hello"; }
	];
};

The service file can then simply reference that path, without having to know the full /nix/store path:

# hello.service
[Unit]
Description=Hello world

[Service]
Type=oneshot
ExecStart=/bin/hello

[Install]
WantedBy=multi-user.target default.target

Now that the service definition is in a separate file, we need to pull it in:

let
	hello-service = pkgs.writeText "hello.service" (builtins.readFile ./hello.service);
in
	...

Reading the file just to write a new copy seems a bit roundabout, but it’s there for a reason - pkgs.portableService expects a derivation rather than a path, so it’ll complain if we try to just pass ./hello.service (we’ll improve this later).

An equivalent and slightly more concise version can be seen in this gist:

let
	hello-service = pkgs.concatText "hello.service" [ ./hello.service ];
in
    ...

Templating

This gist by Github user gdamjan gives an example of templating an input file to use as a service file. pkgs.substituteAll is now deprecated in favour of pkgs.replaceVars (you get a deprecation notice if you try to use substituteAll) so we’ll use that instead:

# flake.nix
{
    inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
     flake-utils.url = "github:numtide/flake-utils";
    };
    outputs = { self, nixpkgs, flake-utils }:
     flake-utils.lib.eachDefaultSystem (system:
         let
          pkgs = nixpkgs.legacyPackages.${system};
          hello-service = pkgs.replaceVars ./hello.service {
              helloExe = "${pkgs.hello}/bin/hello";
          };
         in {
          defaultPackage = pkgs.portableService {
              pname = "hello";
              version = "1.0";
              units = [ hello-service ];
          };
          devShell = with pkgs; mkShell {
              buildInputs = [squashfsTools];
          };
         }
     );
}

# hello.service
[Unit]
Description=Hello world

[Service]
Type=oneshot
ExecStart=@helloExe@

[Install]
WantedBy=multi-user.target default.target

Note the @’s around helloExe - it’s how substituteAll and replaceVars know which text to replace.

Choosing between the two options

The two forms end up being pretty similar, and at runtime have identical results. Ultimately it comes down to preference which one you go for.

It can be useful to have both forms on hand depending on what else you need to do to the file to pull it into your nix definition.

Network binding

Right, enough of the basics - let’s more on to the next step.

On our list is making use of socket activation, so we can (if we want) bind to privileged ports without requiring superuser access or fiddling with CAP_NET_BIND_SERVICE capabilities.

Let’s create a slightly more interesting service, swapping out hello for cowsay, and wrapping it in an inetd-style network layer.

This is a pretty neat option where we configure systemd to bind and listen on a socket for us, then whenever a connection comes in it’ll start a new service instance to handle that connection.

Following the guidance at systemd for Administrators, Part XI, we’ll create a cowsay.socket and a cowsay@.service (note the @, indicating a templated service).

We’ll start with the symlink form to keep things simple:

# cowsay@.service
[Unit]
Description=cowsay

[Service]
ExecStart=/bin/cowsay
StandardInput=socket

# cowsay.socket
[Unit]
Description=cowsay

[Socket]
ListenStream=3000
Accept=yes

[Install]
WantedBy=sockets.target

# flake.nix
{
    inputs = {
     nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
     flake-utils.url = "github:numtide/flake-utils";
    };
    outputs = { self, nixpkgs, flake-utils }:
        flake-utils.lib.eachDefaultSystem (system:
            let
                pkgs = nixpkgs.legacyPackages.${system};
                cowsay-service = pkgs.writeText "cowsay@.service" (builtins.readFile ./cowsay@.service);
                cowsay-socket = pkgs.writeText "cowsay.socket" (builtins.readFile ./cowsay.socket);
            in {
                defaultPackage = pkgs.portableService {
                    pname = "cowsay";
                    version = "1.0";
                    units = [
                        cowsay-service
                        cowsay-socket
                    ];
                    symlinks = [
                        { object = "${pkgs.cowsay}/bin/cowsay"; symlink = "/bin/cowsay"; }
                    ];
                };
            }
     );
}

Immediately we hit our first problem:

nix build
# error: syntax error, unexpected '@', expecting ')'
#        at /nix/store/vqp9bgckrip0vmhchmv6wk1p3sizmw4x-source/flake.nix:17:86:
#            16|         pkgs = nixpkgs.legacyPackages.${system};
#            17|         cowsay-service = pkgs.writeText "cowsay@.service" (builtins.readFile ./cowsay@.service);
#              |                                                                   #                    ^
#            18|         cowsay-socket = pkgs.writeText "cowsay.socket" (builtins.readFile ./cowsay.socket);

Nix really doesn’t like the @ in a path. Luckily this is a pretty easy fix - we can work around it using string interpolation, with a nix expression that’s the @ character we want:

# within flake.nix
let
    pkgs = nixpkgs.legacyPackages.${system};
    cowsay-service = pkgs.writeText "cowsay@.service" (builtins.readFile ./cowsay${"@"}.service);
    cowsay-socket = pkgs.writeText "cowsay.socket" (builtins.readFile ./cowsay.socket);
in ...

A bit ugly, but it’ll do.

Now we’ve got a successful build, but oddly when we test it we’ll find the cowsay@.service file isn’t being copied out:

portablectl attach $(readlink ./result)/*.raw
# (Matching unit files with prefix 'cowsay'.)
# Created directory /etc/systemd/system.attached.
# Created directory /etc/systemd/system.attached/cowsay.socket.d.
# Written /etc/systemd/system.attached/cowsay.socket.d/20-portable.conf.
# Copied /etc/systemd/system.attached/cowsay.socket.
# Created directory /etc/systemd/system.attached/cowsay-.service.d.
# Written /etc/systemd/system.attached/cowsay-.service.d/20-portable.conf.
# Created symlink /etc/systemd/system.attached/cowsay-.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
# Copied /etc/systemd/system.attached/cowsay-.service.
# Created symlink /etc/portables/cowsay_1.0.raw → /home/watchie/.local/share/nix/root/nix/store/y9rhwivprixrbimvlkhidxz4h1lf3bvj-cowsay-img-1.0/cowsay_1.0.raw.

As a result, the socket starts, but as soon as a connection comes in it fails to find the corresponding service and errors out.

What’s going on?

Let’s take a look at our squashfs output again:

nix build
nix develop -c unsquashfs result/*.raw
ls squashfs-root/etc/systemd/system/
# cowsay-.service  cowsay.socket

Hmm, we’ve lost our @ suffix.

Luckily the first issue gave us a hint to what’s happening: nix really doesn’t like @ symbols in derivation names. As it turns out pkgs.writeText is ‘helpfully’ replacing any @ symbols with hyphens to sidestep that issue.

A few attempts with different builders doesn’t result in anything helpful. The closest we get with writeTextDir that places the cowsay@.service inside the derivation folder, which is then valid because the @ doesn’t show up in the derivation name. However packageService then trips up when it tries to write it to the output folder structure.

Adding a symlinks entry instead of putting it in units also doesn’t work - portablectl doesn’t follow symlinks when looking for unit files, and in fact pkgs.portableService takes care to copy the units over instead of symlinking them.

Let’s take a closer look at what pkgs.portableService is doing with the units parameter:

https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/portable-service/default.nix

# units **must** be copied to /etc/systemd/system/
+ (lib.concatMapStringsSep "\n" (u: "cp ${u} $out/etc/systemd/system/${u.name};") units)

So it looks like we need each element in the array to:

  • Have a .name property that renders as a string
  • Itself also render as a string

But how can a value be both an attribute set and a string?

nix coercion trick has a good explanation - an attribute set with an outPath value coerces to a string with the value of outPath when interpolated into a string.

So we can force the output we want: instead of trying to set up a derivation, we can define the attribute set that pkgs.portableService is expecting:

defaultPackage = pkgs.portableService {
    pname = "cowsay";
    version = "1.0";
    units = [
        { name = "cowsay@.service"; outPath = "${./.}/cowsay${"@"}.service"; }
        { name = "cowsay.socket"; outPath = "${./cowsay.socket}"; }
    ];
    symlinks = [
        { object = "${pkgs.cowsay}/bin/cowsay"; symlink = "/bin/cowsay"; }
    ];
};

We still have to do a bit of juggling for the input cowsay@.service file - using ${./.} instead of just ./cowsay${"@"}.socket , which copies the whole folder to a derivation and then pulls out the subfile - but at least it works:

nix build
nix develop -c unsquashfs result/*.raw
ls squashfs-root/etc/systemd/system/
# cowsay@.service  cowsay.socket

Let’s give it another try (note we use reattach this time, which automatically detaches the previous copy)

portablectl reattach $(readlink ./result)/*.raw
# (Matching unit files with prefix 'cowsay'.)
# Removed /etc/systemd/system.attached/cowsay-.service.
# Removed /etc/systemd/system.attached/cowsay-.service.d.
# Created directory /etc/systemd/system.attached.
# Created directory /etc/systemd/system.attached/cowsay.socket.d.
# Written /etc/systemd/system.attached/cowsay.socket.d/20-portable.conf.
# Copied /etc/systemd/system.attached/cowsay.socket.
# Created directory /etc/systemd/system.attached/cowsay@.service.d.
# Written /etc/systemd/system.attached/cowsay@.service.d/20-portable.conf.
# Created symlink /etc/systemd/system.attached/cowsay@.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
# Copied /etc/systemd/system.attached/cowsay@.service.
# Created symlink /etc/portables/cowsay_1.0.raw → /home/watchie/.local/share/nix/root/nix/store/fl5vc31a8cfngn31rgx3sv438r4z420i-cowsay-img-1.0/cowsay_1.0.raw.

echo "hello" | ncat 127.0.0.1 3000
#  _______
# < hello >
#  -------
#           \     ^__^
#            \  (oo)\_______
#                (__)\          )\/\
#                     ||----w |
#                     ||      ||

Pretty neat!

If we take the templating option, we can sidestep the need for the ${./.} trick:

let
    pkgs = nixpkgs.legacyPackages.${system};
    cowsay-service = {
        name = "cowsay@.service";
        outPath =
            (pkgs.writeText "cowsay-template-service" (
                builtins.replaceStrings [ "@cowsayExe@" ] [ "${pkgs.cowsay}/bin/cowsay" ] (
                    builtins.readFile ./cowsay${"@"}.service
                )
            )).outPath;
    };
    cowsay-socket = {
     name = "cowsay.socket";
     outPath = "${./cowsay.socket}";
    };
in {
    defaultPackage = pkgs.portableService {
        pname = "cowsay";
        version = "1.0";
        units = [ cowsay-service cowsay-socket ];
    };
}

Here we use builtins.readFile to read to a string, which doesn’t create an intermediate derivation, and so doesn’t struggle with the @ issue, followed by builtins.replaceStrings (since we’re now dealing with a string instead of a file, as we were with pkgs.replaceVars).

(Note we’re now passing @cowsayExe@ as the matcher instead of cowsayExe, since replaceStrings does a simpler string match than replaceVars did).

Moving on to caddy

As it turns out, all the faff with @ files is for naught, since we’re going to use the other, more common, socket-activation mode for caddy. But it’s good to know we can solve that particular problem if we need to.

pkgs.caddy helpfully provides a service file alongside the binary, so it’s pretty simple to set up a simple portable service:

# flake.nix
{
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
    };
    outputs = { self, nixpkgs, flake-utils }:
        flake-utils.lib.eachDefaultSystem (system:
            let
                pkgs = nixpkgs.legacyPackages.${system};
                caddy-service = pkgs.concatText "caddyplugins.service" [
	                "${pkgs.caddy}/lib/systemd/system/caddy.service"
                    ./caddyplugins.service
                ];
                caddy-socket = pkgs.concatText "caddyplugins.socket" [ ./caddyplugins.socket ];
            in {
                defaultPackage = pkgs.portableService {
	                pname = "caddyplugins";
                    inherit (pkgs.caddy) version;
                    units = [
	                    caddy-service
	                    caddy-socket
		            ];
                    symlinks = [
	                    {
		                    object = ./Caddyfile;
		                    symlink = "/etc/caddy/Caddyfile";
			            }
		            ];
                };
            }
        );
}

# Caddyfile
:3000 {
    # Bind to systemd-provided socket
    bind fd/3
    respond "Hello, world!"
}

# caddyplugins.socket
[Unit]
Description=caddy

[Socket]
ListenStream=3000

[Install]
WantedBy=sockets.target

# caddyplugins.service
[Service]
PrivateTmp=yes
Environment=CADDY_ADMIN=unix//tmp/caddy-admin.socket
AmbientCapabilities=

Here I’ve chosen to use caddyplugins as the system file prefix instead of caddy, to avoid conflicting why any existing caddy files. Otherwise portablectl will refuse to attach the image if it finds a conflict.

To lock down access to the admin API, we also enable two settings:

  • PrivateTmp=yes - sets up a private /tmp and /var/tmp for caddy
  • Environment=CADDY_ADMIN=... - instruct caddy to use this value as the default instead of localhost:2019

And finally we clear the AmbientCapabilities set by the default caddy.service, which otherwise sets AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE - which we don’t need since we’re using socket activation.

Let’s give it a try:

portablectl attach $(readlink ./result)/*.raw
# (Matching unit files with prefix 'caddyplugins'.)
# Created directory /etc/systemd/system.attached.
# Created directory /etc/systemd/system.attached/caddyplugins.service.d.
# Written /etc/systemd/system.attached/caddyplugins.service.d/20-portable.conf.
# Created symlink /etc/systemd/system.attached/caddyplugins.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
# Copied /etc/systemd/system.attached/caddyplugins.service.
# Created directory /etc/systemd/system.attached/caddyplugins.socket.d.
# Written /etc/systemd/system.attached/caddyplugins.socket.d/20-portable.conf.
# Copied /etc/systemd/system.attached/caddyplugins.socket.
# Created symlink /etc/portables/caddyplugins_2.10.2.raw → /home/watchie/.local/share/nix/root/nix/store/bp0wz7iyfzh09s1lff2fs5ns2kjq4afv-caddyplugins-img-2.10.2/caddyplugins_2.10.2.raw.
systemctl enable --now caddyplugins.socket
# Created symlink '/etc/systemd/system/sockets.target.wants/caddyplugins.socket' → '/etc/systemd/system.attached/caddyplugins.socket'.
curl 127.0.0.1:3000
# Hello, world!

Nice!

We can also confirm the admin socket was created by looking for the private /tmp dir created by systemd:

ls /tmp/systemd-private-a939e539ef194cc094a34b62db6e99d8-caddyplugins.service-vQMick/tmp/
# caddy-admin.socket

and checking the logs:

journalctl -xeu caddyplugins.service | grep admin
# Sep 07 18:35:46 smallweb caddy[516514]: {"level":"info","ts":1757270146.9789777,"logger":"admin","msg":"admin endpoint started","address":"unix//tmp/caddy-admin.socket","enforce_origin":false,"origins":[]}

Extending the default configuration

One of the benefits of delivering systemd unit files, instead of docker images, is that the user can configure behaviour flexibly using systemd drop-in files.

Let’s test this out by assigning an environment variable and a custom Caddyfile on the target server - overriding the one delivered with the portable service.

# /etc/systemd/system/caddyplugins.service.d/override.conf
[Service]
# Mount /etc/caddy into the service environment
BindReadOnlyPaths=/etc/caddy
# Set $NAME to User
Environment=NAME=User

# /etc/caddy/Caddyfile
:3000 {
    # Make sure we still bind to systemd-provided socket
    bind fd/3
    # Use the $NAME environment variable
    respond "Hello, {$NAME}!"
}

systemctl reload caddyplugins
curl 127.0.0.1:3000
# Hello, User!

Final step: caddy plugins

Now for the coup de grace: the whole point of all this was that we wanted to deliver custom builds of caddy, complete with additional plugins.

Normally we’d manage caddy plugins with xcaddy, but xcaddy and nix don’t play well together - xcaddy wants to download packages from the internet at build time, and nix runs builds in a network-isolated environment.

Fortunately nixpkgs provides a helper for building caddy with plugins, which performs much the same function: caddy.withPlugins. Let’s give it a try with hairyhenderson/caddy-teapot-module:

# flake.nix
{
    inputs = {
        nixpkgs.url = "github:NixOS/nixpkgs/nixpkgs-unstable";
        flake-utils.url = "github:numtide/flake-utils";
    };
    outputs = { self, nixpkgs, flake-utils }:
        flake-utils.lib.eachDefaultSystem (system:
            let
                pkgs = nixpkgs.legacyPackages.${system};
                caddy = pkgs.caddy.withPlugins {
                    plugins = [
                    "github.com/hairyhenderson/caddy-teapot-module@v0.0.3-0"
                    ];
                    hash = "sha256-+8BLuvRPAjj8bk74UdDpT/E/vg4daAgz8MUa68aYMr4=";
                };
            in {
                defaultPackage = pkgs.portableService {
                    pname = "caddyplugins";
                    inherit (caddy) version;
                    units = [
                        (pkgs.concatText "caddyplugins.service" [
                            "${caddy}/lib/systemd/system/caddy.service"
                            ./caddy.service
                        ])
                        (pkgs.concatText "caddyplugins.socket" [
                            ./caddy.socket
                        ])
                    ];
                    symlinks = [
                        {
                            object = ./Caddyfile;
                            symlink = "/etc/caddy/Caddyfile";
                        }
                    ];
                };
            }
        );
}

(We get the hash value by leaving it empty, running a build, waiting for nix to tell us what it should be, then updating it.)

# Caddyfile
:3000 {
     bind fd/3
     route {
	     teapot
     }
}

Once we’ve built the new image, we can reattach the new version (which performs a detach of the old copy, and mounts the new one), and test it:

portablectl reattach $(readlink ./result)/*.raw
# (Matching unit files with prefix 'caddyplugins'.)
# Created directory /etc/systemd/system.attached.
# Created directory /etc/systemd/system.attached/caddyplugins.service.d.
# Written /etc/systemd/system.attached/caddyplugins.service.d/20-portable.conf.
# Created symlink /etc/systemd/system.attached/caddyplugins.service.d/10-profile.conf → /usr/lib/systemd/portable/profile/default/service.conf.
# Copied /etc/systemd/system.attached/caddyplugins.service.
# Created directory /etc/systemd/system.attached/caddyplugins.socket.d.
# Written /etc/systemd/system.attached/caddyplugins.socket.d/20-portable.conf.
# Copied /etc/systemd/system.attached/caddyplugins.socket.
# Created symlink /etc/portables/caddyplugins_2.10.2.raw → /home/watchie/.local/share/nix/root/nix/store/5nlld9xbc6frfggn4hnr7kyws0ifjpc1-caddyplugins-img-2.10.2/caddyplugins_2.10.2.raw.
systemctl restart caddyplugins

curl -I 127.0.0.1:3000
# HTTP/1.1 418 I'm a teapot

Neat!

Finally we can tear it all down:

systemctl stop caddyplugins.service caddyplugins.socket
portablectl detach caddyplugins
# Removed /etc/systemd/system.attached/caddyplugins.socket.
# Removed /etc/systemd/system.attached/caddyplugins.socket.d/20-portable.conf.
# Removed /etc/systemd/system.attached/caddyplugins.socket.d.
# Removed /etc/systemd/system.attached/caddyplugins.service.
# Removed /etc/systemd/system.attached/caddyplugins.service.d/10-profile.conf.
# Removed /etc/systemd/system.attached/caddyplugins.service.d/20-portable.conf.
# Removed /etc/systemd/system.attached/caddyplugins.service.d.
# Removed /etc/portables/caddyplugins_2.10.2.raw.
# Removed /etc/systemd/system.attached.

Conclusion

What we’ve got here is pretty convenient: with a small amount of nix config, we can deliver:

  • A self-contained package file
  • Built with nix and nixpkgs
  • That we can deliver to any server with portablectl installed
  • Running as many services as we’d like in one package
  • Easily extensible via drop-in files
  • And that’s easy to clean up if we want to remove it

This is only scratching the surface - more complicated setups can deliver timers, socket activation, and multiple services in a single namespace, all by leveraging the systemctl ecosystem.

To see the full code listing, see the repo on GitHub at JaimeValdemoros/portable-service-demo. Each commit corresponds roughly to one change in the article.