Lennart Poettering

Founder + Chief Engineer @ Amutable

Web: https://0pointer.de/blog/
GitHub: https://github.com/poettering
LinkedIn: https://www.linkedin.com/in/lennart-poettering/

Public Key

npub1r9ngmtyw5c9qx49tw9rkty6v89yyq95sys9fzj69v6gy883zfcfsyyts2x

NIP-05 Address

pid_eins_at_mastodon.social@momostr.pink

Profile Code

nprofile1qqspje5d4j82vzsr2j4hz3m9jdxrjjzqz6gzgz53fdzkdyzrnc3yuycpz3mhxue69uhhyetvv9ujuerpd46hxtnfduqs6amnwvaz7tmwdaejumr0dsj0tkw3

Publishing to

relay.damus.io

nos.lol

Author Public Key

npub1r9ngmtyw5c9qx49tw9rkty6v89yyq95sys9fzj69v6gy883zfcfsyyts2x

Show more details

Last Notes

2026-06-02 07:05:29 UTC

- reply

Hence I'll just leave you with some links to docs:
https://github.com/systemd/systemd/blob/main/docs/CREDENTIALS.md#acquisition-from-cloud-instance-metadata-services-imds
https://www.freedesktop.org/software/systemd/man/devel/systemd-imdsd@.service.html
https://www.freedesktop.org/software/systemd/man/devel/systemd-imds.html
And also, if you work for a cloud, and would like your cloud to also be recognized, please talk to us and/or submit a patch for adding it to hwdb. Right now we carry entries for the big three clouds and various smaller ones. Specifically, currently are covered: AWS, Azure, GCP, Hetzner, Scaleway, Tencent, Alibaba.

2026-06-02 07:02:09 UTC

- reply

…order to then allowlist the IMDS service we use source routing with an "fwmark", which is only accessible to privileged code.
The new logic in systemd also provides better security in other ways. For example, before importing system credential systemd-imds will measure them all into a TPM PCR, because they after all are primary, external input for the system's configuration.
There's a lot more I could talk about here, but this post is already long enough.

2026-06-02 06:59:15 UTC

- reply

…typically IMDS information is somewhat security sensitive, since it (sometimes at least) carries cryptographic material and other sensitive and identifying information that should not be accessible to unprivileged payload code, but traditionally is. To remedy this, the new logic in systemd supports locking down direct access to IMDS. For that we can install a "prohibit" route into the IP stack, which ensures all regular programs trying to access IMDS will get EPERM from the socket layer. In…

2026-06-02 06:56:29 UTC

- reply

…provided is not actually intended for systemd-imds, but for example cloud-init, and then just steps away gracefully.
systemd-imdsd is generally useful, even if you don't buy into our system credential concept, as it improves both on reliability and on security over the traditional way to access IMDS. That is because clouds typically heavily ratelimit IMDS access, and refuse to operate if you access it too frequently. systemd-imdsd knows what to do in that case. And it's also because…

2026-06-02 06:53:35 UTC

- reply

…network interfaces that all might or might not be suitable for contacting IMDS.
And then ther's "systemd-imds". This first of all is friendly client tool to "systemd-imdsd.service", for querying IMDS keys from shell scripts or intractively. Secondly, it may run in early boot (initrd typically) to acquire system credentials automatically from IMDS and insert them into /run/credstore/ so that the OS can consume them from there. It's written carefully so that it is fine if the userdata…

2026-06-02 06:51:16 UTC

- reply

…if it is running in a suitable cloud via the hwdb entries. If the answer is yes, it will add two services to the initial boot transaction. First of all there's systemd-imdsd.service. All it does is provide a generic Varlink IPC API fronting the direct IMDS access. It provides both immediate IMDS key access (which is still cloud specific) as well as generic IMDS key access to a vocabulary of well-known keys (which is cloud agnostic). It will do basic caching, and can deal with multiple…

2026-06-02 06:48:36 UTC

- reply

…the minimal amount of information we need to talk to the local IMDS server. This looks like this:
https://github.com/systemd/systemd/blob/main/hwdb.d/40-imds.hwdb
This delivers two things: first of all we auto-detect whether we run in a suitable cloud, and secondly we immediately know enough to talk to IMDS in a generic fashion: the code has no understanding of any idiosyncrasies of the various clouds, only the hwdb has.
There's now a "systemd-imds-generator" which runs early at boot and detects…

2026-06-02 06:46:07 UTC

- reply

…additional device metadata, that cannot be read from the device itself. It originally was introduced to efficiently carry information about all those special keys of the various "multimedia" keyboards from the 2000s, but since then grew into a veritable database with all kinds of facts about hardware. The database is typically keyed by vendor/product info. For the automatic IMDS logic we simply maintain entries that match against the SMBIOS/DMI pseudo-device the kernel exposes, and carry…

2026-06-02 06:43:19 UTC

- reply

…AWS, Azure (and later a few other clouds), and Amutable.
One of the results of this work is the new built-in IMDS support in systemd v261. It's supposed to address the various issues we saw around the current status quo.
First of all, it puts strong focus on auto-detecting the various clouds and doing minimal abstraction over the various flavours of HTTP. For this, we just reused the existing "hwdb" infrastructure of systemd. In case you don't remember: hwdb is systemd's database for…

2026-06-02 06:40:46 UTC

- reply

…are annoyingly different).
IMDS is load bearing for cloud workloads, but it's also full of idiosyncrasies, security issues, reliability issues. And the various client packages are not without issues either: they typically have a major footprint (python in the initrd…), come with a very large codebase (since they have separate plugins for each backend), and a problematic security posture.
A few months ago, we started a project to improve these things, after discussions between folks from…

2026-06-02 06:35:24 UTC

- reply

…client side to the various "instance metadata services" that clouds provide ("IMDS"). IMDS is typically a small HTTP service implemented by the cloud's local, physical system that provides a bunch of JSON fragments both providing information information about the cloud's infrastructure and anything the user themselves provided. (Note that IMDS is what AWS calls this, and some other clouds, but some call it differently, though in essence it's all the same, even if in detail all…

2026-06-02 06:32:24 UTC

9️⃣ Here's the 9th post highlighting key new features of the upcoming v261 release of systemd. #systemd261 #systemd
When operating systems are deployed in the cloud they typically need to be parameterized for first boot, so that they automatically deploy the right configuration, the right payloads, and join whatever infrastructure tooling is appropriate.
This provisioning traditionally happens with the "cloud-init" package (or various alternatives). These tools typically implement the…

2026-05-25 04:44:45 UTC

4️⃣ Here's the 4th post highlighting key new features of the upcoming v261 release of systemd. #systemd261 #systemd
On many servers and embedded devices serial console console access on boot is absolutely essential. For that reason, firmwares (UEFI), boot loaders, and the kernel itself all support sending their boot time output to a serial port, and receiving input from it.
Configuring serial consoles in the boot loader and Linux is kinda painful though: you need to know some…

2026-02-13 04:26:58 UTC

- reply

it's so laughable, so cringe that they brag about "pull request diff performances". These days github is barely usable anymore for patch reviews. The pr review tool is broken and slow. The two distinct versions they offer are in competition which one is more broken. Half the patch context links they have never get you to the right place, and they collapse anything by default that's larger than a couple of dozens of lines. So many clicks for everything. And the slow UI reaction times...

2026-01-27 16:34:19 UTC

I don't often share Phoronix articles, but this one I will:
https://www.phoronix.com/news/Amutable

2025-11-25 15:30:55 UTC

6️⃣ Here's the 6th post highlighting key new features of the upcoming v259 release of systemd. #systemd259 #systemd
Here's a short one: systemd v259 will compile fine with musl libc, out of the box.
Sounds great? Well, it's not as great as it might sound to some. musl has quite some limitations compared to glibc: the primary one is that there's no Name Service Switch (NSS) support. That's the subsystem that allows systemd to make domain names, user names, groups names resolvable via…

2025-11-20 07:43:09 UTC

- reply

rpm and dpkg extract the dlopen deps when building packages and turn them into Suggests/Recommends packaging deps.

2025-11-20 07:43:04 UTC

- reply

static libs doesn't work for this. The main reason why systemd's size footprint is actually quite OK for everything it does, is primarily because we do not use static linking, and hence duplication of code in many ELF objects, but are pretty good and minimizing that and placing any shared code in a common object file.
And if you export something to the public you must keep ABI stable, no matter what.

2025-11-19 15:41:00 UTC

- reply

it used to be split out a long time ago. it was a fricking nightmare of deps and came with so many restrictions, because ELF cannot distinguish between public APIs to everyone, and public APIs towards a certain set of of libraries. i.e. anything a library exports is *always* public for anyone, thus you can never provide a simple helper for your own higher level libraries only, you must *always* commit to it's ABI stability if you do.
Hence, sorry, fuck no. Never again.

2025-11-19 08:02:23 UTC

- reply

And one last thing: what about those last two deps? i.e. openssl and libcrypt?
Yes, we have plans to turn those into dlopen() deps too. Hopefully in v260.
And once we achieve that things are going to be fun, because systemd will start to have a smaller minimal dep footprint than certain other "lightweight" init systems. For example, that dependency hog s6 is currently at 3 shared library deps beyond libc. So wasteful! And it doesn't even do a fraction of what systemd does...

2025-11-19 07:55:16 UTC

- reply

Oh, and it's not just about footprint actually, it's also about security: by ensuring that dynamic libraries only get loaded when they are actually used, we make it harder for exploits such as the openssh/xz incident last year to take place, as compile time deps no longer translate to runtime deps 1:1, and thus awful concepts such as gcc constructors lose their negative impact a bit.

2025-11-19 07:51:35 UTC

- reply

No we don't, because years ago we devised a spec for embedded dlopen() based weak dependencies in ELF binaries. systemd binaries use that comprehensively for everything they dlopen(), to the point that libsystemd-shared.so now declares a whopping 26 of them. All relevant package managers have since been updated to read this metadata, hence in this regard we have not regressed.

2025-11-19 07:49:29 UTC

- reply

…(which is a library a myriad of other programs link to) can have a tiny dependency footprint too.
Or in other words, if you try to build a minimal container image, then it's certainly not systemd that's going to be the dependency hog that blows up its size.
Now, you might wonder about package managers: they tend to automatically generate package dependencies from shared library dependencies, so if we get rid of those, are we breaking all package managers?

2025-11-19 07:47:21 UTC

- reply

It's really about footprint: with one exception you don't really need any of these deps in certain limited setups: when you run systemd inside of a container for example.
You wonder about that one exception? That's util-linux' libmount API. We actually need it for the service manager to run, so even though it's now a weak dependency it really isn't if you run the main systemd binary. You might wonder why bother then? The reason is to ensure that libsystemd.so…

2025-11-19 07:44:11 UTC

- reply

Yes, they indeed, at least kinda. With v259 we converted almost all our runtime dependencies to use dlopen() instead of explicit shared library linking. We started this process back in v247 with some "leaf" libraries, but with v259 we moved 7 more libraries over, so that in effect only two non-libc libraries remain that we link regularly to: libcrypt (aka the crypt() password hashing API) and libcrypto (aka OpenSSL, and the latter pulls in libz).
You might wonder why we do this?

2025-11-19 07:39:54 UTC

- reply

If you look at this closely, you should notice that the deps listed there now are only: systemd's own shared libraries, glibc/gcc stuff (libc, libm, ld-linux, libgcc_s), and only three other things: libcrypt, libcrypto and libz.
But how can this be? systemd used to link against pam, libselinux, tpm2-tss, lbacl, libblkid, libmount, libbpf, libcryptsetup, libaudit, libkmod, libpcre2, libp11-kit, libseccomp, liblzma, libzstd, liblz4, …
But now they are all gone?

2025-11-19 07:33:18 UTC

- reply

libcrypt.so.2 => /lib64/libcrypt.so.2 (0x00007f4511fcb000)
libcrypto.so.3 => /lib64/libcrypto.so.3 (0x00007f4511000000)
/lib64/ld-linux-x86-64.so.2 (0x00007f451250e000)
libz.so.1 => /lib64/libz.so.1 (0x00007f4512384000)
And that's it.
Do you notice something about this output?
No? I'll give you a hint, it's not about what's in it, it's really about what is not in it.

2025-11-19 07:32:28 UTC

- reply

linux-vdso.so.1 (0x00007f451250c000)
libsystemd-core-259-rc1.so => /usr/lib64/systemd/libsystemd-core-259-rc1.so (0x00007f4512000000)
libsystemd-shared-259-rc1.so => /usr/lib64/systemd/libsystemd-shared-259-rc1.so (0x00007f4511800000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f451249d000)
libc.so.6 => /lib64/libc.so.6 (0x00007f451160c000)
libm.so.6 => /lib64/libm.so.6 (0x00007f45123a9000)

2025-11-19 07:32:03 UTC

2️⃣ Here's the 2nd post highlighting key new features of the upcoming v259 release of systemd. #systemd259 #systemd
If you have systemd v259 on your system, and you run "ldd /usr/lib/systemd/systemd" (i.e. query the shared library deps of the systemd service manager binary), then you get the following output:

2025-09-01 09:06:16 UTC

4️⃣9️⃣ Here's the 49th post highlighting key new features of the upcoming v258 release of systemd. #systemd258
One of the key features of systemd from day 1 on is socket activation, i.e. a mechanism where systemd binds sockets on behalf of services, watches them and only activates the services themselves later, possibly only at the moment they are actively used.
This has various benefits, for example reduces ahead of time cost of running a large number of services (which improves boot times).

2025-08-19 08:36:59 UTC

- reply

Classic Linux is a multi-user OS with security isolation between users. This means users should not be able to sniff on other user's keyboard or mouse input, or read raw data off block devices bypassing file access restrictions and so on. Because of all that we cannot just wildcard allow unpriv users raw access to USB devices: we must enforce access restrictions.

2025-08-18 04:47:26 UTC

- reply

…logic, i.e. giving any local user with a session in the foreground access to them.
And that's already the whole post for today.

2025-08-18 04:46:22 UTC

4️⃣0️⃣ Here's the 40th post highlighting key new features of the upcoming v258 release of systemd. #systemd258
It's a quick one: Android USB debugging might not be an official standard, but it's implemented by a myriad of devices. Previously accessing Android USB debugging interfaces from regular, unprivileged programs required installation of manual udev rules.This should now be a thing of the past, we now match these interfaces out of the box and make them accessible through the "uaccess"…

2025-07-14 07:24:24 UTC

3️⃣2️⃣ Here's the 32st post highlighting key new features of the upcoming v258 release of systemd. #systemd258
systemd-repart is systemd's dynamic repartitioner and disk image (DDIs) builder. One of its strengths is in the area of cryptographic protection: the ability to generate Verity enabled file systems + signing them, and including all that in the final image (file system + Verity data + signature for the top-level root hash).

2025-06-16 07:08:38 UTC

1️⃣7️⃣ Here's the 17th post highlighting key new features of the upcoming v258 release of systemd. #systemd258
In systemd we focus a lot on the integrity of the OS. Among various other things this means strong support for Verity protected, immutable file systems. You can run the OS from one, and you can run services from them, as well as containers.
We generally recommend placing Verity protected file systems in DDIs ("Discoverable Disk Images"), …

2025-06-01 06:48:42 UTC

7️⃣ Here's the 7th post highlighting key new features of the upcoming v258 release of systemd. #systemd258
systemd is at its most basic a service manager, i.e. it runs programs, in a resource managed, security sandboxed way, properly ordered, and starts the system that way and keeps it running.
The focus for this kind of service management is really on services that are started no matter what, regardless of the resources available, because the underlying assumption…

2025-05-21 12:46:56 UTC

It's that time again! The systemd v258 release is coming closer. Let's restart the "what's new" series of posts for this iteration! Hence:
1️⃣ Here's the 1st post highlighting key new features of the upcoming v258 release of systemd. #systemd258
As most of you probably know "systemctl start" is how you manually start a systemd unit. Starting a unit can fail, and systemd tracks that for you and tells you this. When you encounter such a failure the next thing you'd typically do…

2025-02-26 08:59:25 UTC

I wrote some docs regarding how we think a modern Linux-based OS should boot and find the root file system, and the 3 foundational components of it.
For now, it's only available in systemd's gid main branch, but on the next release it will also be visible on https://systemd.io/.
For now:
https://github.com/systemd/systemd/blob/main/docs/ROOTFS_DISCOVERY.md
This is vastly simpler than traditional Linux boot with Grub and all that stuff, and the security properties are so much nicer.

2025-02-10 13:04:28 UTC

Fun little thing I have been working on: teach systemd to boot directly into a disk image downloaded via HTTP within the initrd.
In v257 systemd learnt the ability to download disk images at boot via systemd-import-generator, both DDIs and tarballs, and place them in /var/lib/machines/, /var/lib/portables/, /var/lib/confexts, /var/lib/extensions/. The goal was to provide a way to provision any of these resources automatically at boot. But now that we have this, we can take it a step further:

2024-12-16 08:49:03 UTC

3️⃣5️⃣ Here's the 35th post highlighting key new features of the current v257 release of systemd. #systemd257
systemd-homed is the user/home area managed service of systemd. It's designed to provide very secure home directory management on Linux OSes. One fundamental idea is that the user's provided unlock credential (password, FIDO token, PKCS11 token) are actually what the encryption key for the home directory is derived of. This is of course fundamentally different from traditional UNIX, …

2024-11-25 09:05:08 UTC

1️⃣8️⃣ Here's the 18th post highlighting key new features of the upcoming v257 release of systemd. #systemd257
With the systemd project we are trying to push distributions to adopt a "Hermetic /usr/" model. This means that the vendor OS resources are monopolized within the /usr/ hierarchy, and that it's sufficient to mount a distro /usr/ tree into an otherwise empty root file system in order to boot it up.
To make this work, a skeleton directory hierarchy outside of /usr/ must…

2024-11-07 11:06:22 UTC

- reply

… but actually is just some malware that exfiltrates the password you type in?
Since this kind of attack scenario is not new, many OSes provide a "SAK" concept, which stands for "Special Attention Key". The idea is that there's a special key combination you can hit first, which no web page, or web browser, or app, or even desktop environment could possibly hook into that always brings you back to your *real* login screen, regardless where you are.

2024-11-07 11:02:19 UTC

7️⃣ Here's the 7th installment of posts highlighting key new features of the upcoming v257 release of systemd.
The graphical login prompt you see when your computer boots up is a sensitive UI: typically, when starting to work, without much thinking you'll type in your username and password, expecting it to log you in and provide you with your desktop session. However, what if someone just opened a website in a browser in full screen mode with contents that just *looks* like your login screen, …

2024-10-31 21:06:46 UTC

- reply

If you ask me, it's a fundamental requirement for any modern Linux-based OS to provide boot time integrity and as baseline provide unattended disk encryption bound to it. To make this happen, we added two essential TPM policy concepts to systemd-cryptenroll/systemd-cryptsetup:
1. Signed TPM PCR policies allow locking a disk to a public signing key of an OS vendor, ensuring that disks can only be unlocked if an OS signed by said vendor is booted.

2024-10-31 21:03:11 UTC

2️⃣ Here's the 2nd installment of posts highlighting key new features of the upcoming v257 release of systemd.
In the past year and a bit I spent a lot of time on boot integrity (i.e. boot-time TPM measurements and policies built on top of them) of Linux, covering the boot from the boot loader (systemd-boot), over the UKI EFI stub (systemd-stub) through the initrd into early regular userspace, and then locking disk encryption to it.

2024-10-28 09:43:05 UTC

- reply

Two caveats though: the concept is not universal: it's a Linux thing, and it requires kernel 6.9 or newer and a 64bit architecture. On 32bit the inode number range is too small to provide unique IDs.
To properly check if the feature is available allocate a pidfd, and check if statfs() reports a .f_type field of it being 0x50494446. Also verify if sizeof(ino_t) is >= 8.

2024-10-28 09:40:35 UTC

- reply

It took a long time, but thanks to @npub19qk…d5gn after all those years the limitations of UNIX pid_t are addressed! Thanks, Christian!

2024-10-28 09:39:02 UTC

- reply

I think the pair of PID and pidfd inode number would be great to support in the various tools that currently deal with PIDs. For example, I filed an RFE bug against util-linux' kill tool to add just that:
https://github.com/util-linux/util-linux/issues/3252

2024-10-28 09:37:51 UTC

- reply

… when we pass around information about processes via IPC we have started to do so via the triplet pid, pid inode, boot id.
And I'd recommend everyone dealing with low-level process management to do the same.

2024-10-28 09:36:46 UTC

- reply

If you want a world-wide unique identifier for a process it makes sense to combine the pair of pid_t and pidfd inode number with the system's boot ID (i.e. /proc/sys/kernel/random/boot_id). This triplet is awesome, because for the first time we can uniquely identify a Linux process, globally in this universe.
In systemd we are making use of this heavily now: internally we always store a triplet of pid, pidfd, pidfd inode for referencing processes we manage and…

2024-10-28 09:34:33 UTC

- reply

To query the inode number from a pidfd, you use a simple fstat() call, and look at the .st_ino field.
There's currently no way to get from a pidfd inode number directly to a process however. Hence, for now you always have to pass around a combination of classic PID and the new pidfd inode number. This can be safely and correctly be turned into a pidfd: 1. first acquire a pidfd from the PID via pidfd_open(). 2. Then fstat() the fd, and check if .st_ino matches the expected value.

2024-10-28 09:31:56 UTC

- reply

These inode numbers are (at least on 64bit archs, i.e. anything modern) unique during the entire runtime of a system. And that's fantastic: there's finally a way how you can race-freely reference a process, with the ability to pass it around over any form of IPC, without risking that it suddenly starts to refer to some unintentended other process.

2024-10-28 09:29:59 UTC

There's a feature added to Linux 6.9 that I think people should become more aware of: there's finally an identifier for processes that doesn't wrap around as easily as UNIX pid_t PIDs do: the pidfd file descriptors have been moved onto their own proper file system (pidfs), which enabled at the same time unique inode numbers for them.

2024-08-27 08:02:27 UTC

- reply

…you really have to.
(I am trying to do my part on this of course, i.e. in systemd we measure a lot of things during boot now, and our FDE logic is hooked up with it.)
[That all said, I think SB might have some value if you enroll your own keys, which however can only work on very specific hw, and in VMs, but is probably not a solution realistic for general purpose PCs]

2024-08-27 08:00:23 UTC

- reply

…it is is "democratic", in the sense that anyone can do this without having to get their keys into some centralized keyring.
Hence, to me it implications of SB are simply not worth it, it brings very little to the table security wise, but creates massive headaches on deployment. MB otoh actually provides a high level of security, and you don't have to ask anyone to put together your own policies.
Hence if you ask me: focus on making MB a thing on Linux, and bother with SB only to the level…

2024-08-27 07:57:09 UTC

- reply

Much more interesting is Measured Boot when tying disk encryption to it. Various OSes, including Windows have been supporting this since about forever. And it's so much better: it basically makes no restrictions on what you can run on your PC. All it enforces is: my encrypted disk can only be decrypted if the OS of my choice is booted in the version of my choice. And that's a *way* more powerful concept, because it is *focussed* on your installation, because…

2024-08-27 07:53:46 UTC

Noone asked me, but if you are curious what my take on the recent sbat/SecureBoot kerfuffle is, I'll let you know anyway:
Frankly, I find SecureBoot ultimately pretty uninteresting tech. It casts a very wide net: it basically is a politically charged global allowlist, yet is useful as a very very lose denylist only, because it necessarily contains so so so much stuff. I think the value for security is relatively limited, because it it attempts to be universal, and hence can never be focussed.