I run a small VPS that runs all sorts of odds and ends - test services I’ve playing around with, static pages I’m hosting, etc.

The main entrypoint from the outside world is caddy - caddy listens on ports 80 and 443, inspects incoming connections, then decided what to do with them.

Thanks to HTTPS, all these services can run off the one VM all sharing the same IP - each service gets its own domain name, then Caddy receive all the connections, check the domain name being requested, and dispatch the right action - anything from:

Serving files directly
Generating hardcoded responses
Proxying to a service running on a given port
to loads of additional options through its plugin mechanism

What are we building?

I came across a suggestion to configure Caddy to return the client’s IP address when you got to a particular URL, which I thought would be neat to run on my own infrastructure.

The basic suggestion involves just using Caddy’s templating directive to return the IP directly:

ip.{$DOMAIN} {
    templates
    header Content-Type text/plain
    respond "{{.RemoteIP}}"
}

This saves you having to go to websites like whatismyipaddress.com to find out your IP address, and instead rely on something you control that isn’t trying to sell you something.

Then I remembered that I recently looked at geo-locating IP addresses, and thought it would be even more neat if as well as getting an IP address back, you’d also get back what region you seem to be in. How hard can it be?

The approach

Having a single Caddy server fronting a bunch of services works great - however, each time I add another service to my server, that service then sits there consuming some (small) amount of CPU and RAM.

Wouldn’t it be better if the service shut down when it wasn’t being used, and started up-on demand whenever a request came in?

That’s exactly what we’re going to do.

Enter systemd

systemd is a lot of things, most of them too big to go into in this article. For our usecases it’s a service manager, dealing with starting up services, dealing with dependencies between services, restarting them if they fail and generally managing their lifecycles.

While system systemd is mainly known for starting up services at boot time, it’s also got slightly lesser-used features that can be used to trigger services in a flexible way:

At boot up time, with dependency ordering
Repeated or one-of timers
In response to files being created or modified
When connections come in (which is what we’re using)

Socket activation

In this case, we’re going to use something called socket activation. We’ll let systemd listen for connections for us, and when a request arrives it’ll:

Check if the service is running
If not, delay the connection until the service starts
Pass the request to the service

The exact details depend on how we configure systemd. There’s a great overview at systemd for Developers I and systemd for Administrators, Part XI, but effectively there’s two modes, which I’ll call ‘per-connection’ and ‘single-listener’:

In the single-listener mode, systemd waits for the first connection over a socket. It then starts up the service, passes the whole socket to the service, then the service is responsible for receiving the connection and subsequent connections. After some amount of inactivity, systemd may shut down the service
In the per-connection mode, systemd keeps hold of the listener. When any connection arrives, it creates a new instance of the service and lets it handle that specific connection until it’s done and shuts down. If N connections arrive, systemd starts N copies of the service, one per connection.

The first mode can be more efficient if you have lots of connections coming in, since you have one copy of the program handling all of them.

However if you only have connections coming in infrequently, the second might be much simpler to write from a code point of view. You can even have systemd bind the connection to STDIN and STDOUT, so the program doesn’t need to know anything about connections and just works in terms of reading input and producing output. This is sometimes referred to as inet-style, as it’s how inetd used to work.

systemd for Administrators, Part XI already has a great example of how to set up an inet-style service, so let’s focus on the single-listener mode.

Single-listener mode

Since we’re writing our service in Rust, we’ll use the listenfd crate to handle receiving the socket from systemd. The reason for a specialised crate is that most socket usecases assume the service is creating the socket, whereas in this case systemd creates it for the service, and then passes relevant information in via environment variables. listenfd handles reading these environment variables, grabbing the socket and clearning the variables so no other bit of code tries to grab them.

We can define the initial boilerplate:

use listenfd::ListenFd;

fn main() -> anyhow::Result<()> {
    let mut listenfd = ListenFd::from_env();
    let _socket = listenfd.take_tcp_listener(0)?;
    Ok(())
}

and then test it out using systemfd:

systemfd --no-pid -s tcp::5000 -- cargo run
# ~> socket 127.0.0.1:5000 (tcp listener) -> fd #3
#     Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.23s
#         Running `target/debug/socket-geoip`

Once we’re happy that’s working, we can create our initial user-level systemd files. Assuming we’re going to cargo install the service, it will normally end up at ~/.local/bin.

# ~/.config/systemd/user/geoip.service
[Service]
ExecStart=%h/.local/bin/socket-geoip

# ~/.config/systemd/user/geoip.socket
[Socket]
ListenStream=127.0.0.1:5680

[Install]
WantedBy=sockets.target

This instructs systemd to set up a listener at port 5000 (any other free port would do), and on any connection pass the listener to socket-geoip to take over.

Communicating between Caddy and our service

Looking at Caddy’s reverse_proxy options, we see that Caddy supports two ‘transports’:

http
fastcgi

HTTP would be the obvious choice - there’s libraries for it in pretty much every language - but comes with a bit of overhead. We need to pick a HTTP framework (of which there are many), decide if we want to use async or not, etc. etc.

FastCGI on the other hand is a much simpler protocol, designed specifically to provide simple communication between a reverse proxy like Caddy and a backend application. I’ve been itching to play around with it for a while, and combining it with systemd’s socket activation feels like the perfect usecase!

Conveniently there’s a simple-looking crate available: rust-fastcgi. We can pull this in and combine it with our socket activation logic, passing the socket from listenfd to fastcgi to take over.

First let’s write a really simple implementation that just replies ‘Hello, world!’. Note since fastcgi takes a raw socket, we switch the ListenFd call to take_raw_fd.

use std::io::Write;

use listenfd::ListenFd;

fn main() -> anyhow::Result<()> {
    let mut listenfd = ListenFd::from_env();
    let socket = listenfd.take_raw_fd(0)?.unwrap();
    fastcgi::run_raw(
        |mut req| {
            write!(
                &mut req.stdout(),
                "Content-Type: text/plain\n\nHello, world!"
            )
            .unwrap_or(());
        },
        socket,
    );
    Ok(())
}

Once we’ve got it installed, we can start the service with systemctl --user start —now geoip.socket, and update out Caddyfile to point to it:

ip.{$DOMAIN} {
        reverse_proxy / localhost:5000 {
                transport fastcgi
        }
}

Let’s see if it’s worked:

curl https://ip.$DOMAIN
# Hello, world!

Nice!

Business logic

Now we’ve got Caddy and our service talking to each other, with our service providing some simple output. Now we just need to get the IP data!

In order to do an IP lookup, our service needs two things:

A path to a GeoIP database, shared by all requests
An IP address to look up, set per-request

For the former, we can download one of the databases from [🔦spotlight] geo-locating IP addresses, and put it in our home directory. We can then set an environment variable in our systemd file to tell the service where to find it:

[Service]
ExecStart=%h/.local/bin/socket-geoip
Environment=DB_FILE=%h/GeoLite2-City.mmdb

For the per-request IP, we need Caddy to pass through the caller’s IP as part of the fastCGI request. This is easy enough to add:

ip.{$DOMAIN} {
        reverse_proxy / localhost:5680 {
                transport fastcgi {
                        env REMOTE {remote}
                }
        }
}

Now we just need to implement the logic in our service:

use std::io::Write;
use std::net::SocketAddr;
use std::sync::Arc;

use listenfd::ListenFd;
use maxminddb::geoip2::City;

fn main() -> anyhow::Result<()> {
    let mut listenfd = ListenFd::from_env();
    let socket = listenfd.take_raw_fd(0)?.unwrap();
    let reader = Arc::new(maxminddb::Reader::open_readfile(
        std::env::var("DB_FILE").unwrap(),
    )?);
    fastcgi::run_raw(
        move |mut req| {
            let addr = req.param("REMOTE").unwrap();
            let mut stdout = req.stdout();
            let _ = write!(stdout, "Content-Type: application/json\n\n");
            let Ok(addr) = addr.parse::<SocketAddr>() else {
                return;
            };
            if let Ok(Some(city)) = reader.lookup::<City>(addr.ip()) {
                let _ = serde_json::to_writer_pretty(stdout, &output);
            }
        },
        socket,
    );
    Ok(())
}

Now we need to make sure our changes are applied:

cargo install --path . to update the binary
systemctl --user daemon-reload to let systemctl pick up the changes

Let’s give it a try:

Seems to work!

Shutting down

Now we’ve got a service that starts up on-demand when a request comes in.

However, once the service starts up, it’ll run indefnitely - systemd has handed over the connection to the service, so it has no way of knowing when the request is done and whether there’s more requests coming in.

To implement this, we’ll add a fairly crude idleness check:

Take a timeout $TIMEOUT_SECS as an environment variable
Track a ‘ticker’ of requests as an AtomicU64
On each request, increment the ticker
Spawn the background thread that:
- wakes up every $TIMEOUT_SECS seconds
- checks if the ticker has incremented
- if it hasn’t, call std::process::exit(0)

static TICKER: AtomicU64 = AtomicU64::new(0);

fn main() -> anyhow::Result<()> {
    ...
    if let Ok(timeout_secs) = std::env::var("TIMEOUT_SECS") {
        let timeout = Duration::from_secs(timeout_secs.parse().unwrap());
        spawn(move || {
            let mut last_ticker = 0;
            loop {
                sleep(timeout);
                let ticker = TICKER.load(Ordering::Relaxed);
                if ticker == last_ticker {
                    eprintln!("idle, exiting");
                    eprintln!("{ticker} requests served");
                    std::process::exit(0);
                }
                last_ticker = ticker;
            }
        });
    };

    fastcgi::run_raw(
        move |mut req| {
            TICKER.fetch_add(1, Ordering::Relaxed);
            ...
        }
    )
}

We can then update our service file to include the timeout - in this case we’ll pick 1 minute, or 60 seconds:

[Service]
ExecStart=%h/.local/bin/socket-geoip
Environment=TIMEOUT_SECS=60
Environment=DB_FILE=%h/GeoLite2-City.mmdb

And indeed, we can see the service shuts down:

journalctl -f --user -u geoip.service
# Aug 21 10:39:50 smallweb systemd[864]: Started geoip.service.
# Aug 21 10:42:50 smallweb socket-geoip[732194]: idle, exiting
# Aug 21 10:42:50 smallweb socket-geoip[732194]: 13 requests served

Roundup

This structure feels pretty satisfying - by leveraging features in Caddy and systemd, we’ve managed to drastically simplify the logic our service needs to deal with and the resource consumption while idle.

Recapping:

Caddy listens for incoming connections, inspects them, and connects to the right backend service to ask for a response
systemd watches for connections coming in to backend services, and when a request comes in dynamically starts up the corresponding service
Caddy and the backend service talk using the FastCGI protocol
The service stays awake as long as it needs to process that and any further requests, and then shuts itself down

Since Caddy and systemd are the only things staying awake, adding additional services doesn’t contribute to resource usage - at least, until they start getting requests coming in.

You can find the full code listing, along with a bunch of improvements I’ve made since this article, on my Github: JaimeValdemoros/socket-geoip.

What are we building?#

The approach#

Enter systemd#

Socket activation#

Single-listener mode#

Communicating between Caddy and our service#

Business logic#

Shutting down#

Roundup#