Good morning everybody. My name is Tobias
Deiminger.
I'm a software engineer at Linutronix and
I'm also occasional Debian contributor.
It's the first time for me to meet you in
person.
I'm very happy about that. And I tell you
about how embedded device
vendors can use Debian to create immutable
systems.
And a special special thing about this
flavor of immutable system will be that
it's using systemd root and shim.
Now, let's first define what is an
immutable system.
And
I think we could say that it's an
operating system where the root file
system is write-protected and updates are
applied by atomically replacing the whole
root file system. Now, we have seen
in the talk by Jörg that LearnStick is
such a system.
I would say if we enable DM Verity and the
TPM support and remote attestation,
this fully is this definition.
So what I will go on with my talk.
I think there are.
Differences here. What I will show you is
more suited for embeddable systems, but
it's comparable.
There are a lot of examples out there of
immutable systems, for example, or your
Android phones or immutable systems.
And now we will see how we can build this
with almost unmet modified Debian.
Why do we want immutable systems at all?
It's a very robust mitigation for certain
threads.
Every thread where you need, where the
attacker needs persistence, where the
attacker needs to modify or swap your disk
or
compromise running systems, then install
some boot kit, for example, and have
something to
exploit during the next boot.
This is also important for vendors that
have to comply with security standards
because almost every security standard is
wanting authenticity and integrity in some
form.
Most notable maybe the CRA. They want it
in
different rigor regarding your use case,
but the high-level requirement of
authenticity and integrity is always there
and this is something that Immutable
System addressed very well.
And of course for embedded systems it's
very convenient um because
at least the flavor that I will show you
is self-verifying, starting from uh a
trusted uh platform uh anchor uh sealed in
hardware. And for for embedded devices
that are in the field, remote
attestation may not be an option.
Maybe it's it is, I I'm not yet sure, I
have not seen such such cases in the
embedded world.
Yeah, and because of that uh what we are
doing here is a kind of verified boot.
This means that starting with uh
trusted firmware. We load some operating
system and bootloaders and before
executing them we verify the signature of
this operating system and
bootloader images and only continue if the
signature verification is successful.
If not we will immediately stop the boot
and do nothing.
This is the big difference to measured
boot where we would just continue and some
remote party would then say, hey, there's
something not
fine.
We have to do some action.
And then we are not yet finished.
This is only the boot part, but now to get
the full immutable system, we have to
chain this trust into the root file
system.
And for that, we can
we will see this in a minute. We can tie
the hash tree of the root file system to
Linux, to the Linux kernel, so that it
will only read the root file system
successfully if the hashes match.
And the hash itself is part of the secure
boot.
That's important. That's how we chain this
together.
This idea is there for a long time and I
think people from the civil infrastructure
project did a lot of good work there and
made this popular.
You could look at their presentations.
But there were shortcomings that I try to
address now.
For example, one thing that was missing
was rollback protection, meaning if you
have some
version one of your software that is
signed by your signature,
but it turns out that it contains a
vulnerability and you have already version
two in the field. Now some some attacker
could come and try to
install version 1 again just to get this
exploitable version.
And how do you protect it? It's validly
signed, so you need some other mechanism
to prevent this, and this is rollback
protection.
This will be possible in this setup.
And then at least in the work we did at
Linutronix, it turned out
that the former implementation is a bit
tricky to do the trust provisioning.
This is now also a lot more streamlined.
Also the boot, the system D boot parts are
a lot more streamlined.
We don't have to hack that much ourselves
now.
And this is
the actual proposal on a single map that
shows the full boot flow.
Let's let's explore what we have there.
We start with UAV I would say.
UAV is
is good because it's everywhere and we can
rely on a hardware trust store if we have
UEFI. Then if we look one left at this
picture, then we
see we could also have UEFI on embedded
systems that are not x86
because U-Boot has UEFI emulation.
So if we need that, we can We can first
boot U-Boot, enable the UEFI emulation,
and then just continue the scheme that is
seen here.
I did my studies of this system with QEMU
x86, but Javi, who
is also there,
did the same for an ARM 64 architecture,
and it's really much the same.
It worked, and it's much the same.
UEFI will start, it will load Shim.
Shim is a tiny bootloader that is
originally invented to
get it signed by Microsoft, but you can
embed your own certificate.
For example, Debian is doing this.
Debian is embedding its own certificate
into Shim so that every further
bootload can then be verified using your
own certificates.
I do not use shim here for the purpose of
this certificate chaining, but for the
purpose of SBAT revocations. This is the
rollback protection.
Embedded vendors probably don't care that
much about this Microsoft signings.
They have the system under their control
and sign everything by themselves.
But rollback protection is unsolved and
shim can solve this in a in a good way.
And then the next stage will be systemd
boot.
It's a smaller alternative to GRAP, which
has also the convenience that it can
handle AB redundancy, which is then the
next part.
I don't know how this would work in GRUP.
Does anybody of you know? Can GRUB do AB
redundancy?
You can do that so yeah, because you can
you can do the GRAP configuration.
Yeah, so keep in mind GRAP might be an
alternative here.
Then we see the the fork into two possible
paths.
Path.
The the one path is
Partitions for the the A system, and then
we duplicate everything and have
partitions for the B system. And why are
we doing that?
Because we are expecting over the year
updates, and that can fail.
And if they fail, we have some undefined
state, and we need need to fall back into
the formally working system. And so we
need some bootloader that is capable of
figuring out what was the last working
state.
And if we have to try a new state.
And this is done by system debut here.
And then
we see UKIs.
This is Linux wrapped in FI, basically.
This is a binary that that has a FE stub,
but then
executes the Linux kernel with the
ordinary x86 boot protocol.
And
Again, this is a system deboot feature
that I that is probably not in GRAP, I
don't know. I would be surprised.
It's UKI add-ons. You can not only have
one main UKI, but you can merge
several UKIs together. And now, since we
are supplying the root
hash of the DM Verity partition here as
command line of UKI, this enables
us to do
separate rollback protection only for the
root file system.
This is the Probably the main point of
this presentation.
We can
have the root hash standalone signed and
SBAT verified.
Okay, and
then on the right side we see what the
user is working with.
We have
the now immutable X4 partition, which is
protected by the Mverity.
We have an overlaid FS because very
probably we will want to have m
modifiable configuration or other
application data.
So we do use DMcrypt for that and seal the
DMCrypt
key slots with TPM.
And that's probably it for that overview.
Now let's build such a system.
How do we start? Since we are building on
Debian,
we first need to build our packages.
We don't want to do it Yocto style and
build everything ourselves, but we want to
reuse what's there from Debian.
And this works for, let's say, 99% of the
packages.
We can just pull them upstream from the
Debian archive.
But for that packages that need signing,
we we need some special handling.
And this in this case our shim system
deboot and another custom package
that is called Unify Kernel. It contains
the Linux kernel and a
special precar created initial ROMFS
and wraps it into into a UKI. And it is
also signed.
And
we now how can a vendor do that signing?
Debian solves this by having this signed
template
packages. I'm not sure if you have seen
them.
They con contain basically a recipe, what
files of an unsigned package
need signing and what type of signings I
need and then they also contains a rules
file how to assemble the the signatures
into the unsigned
binaries. And we we can reuse that here.
We a few days ago we have open sourced
OpenSIC Cub.
This is a small tool that allows you to
reuse the signing templates from Debian.
and sign them with your own keys.
That are in HSM. This was an important
requirement, so everything is based on
PKCS 11 here.
Yeah, and it supports not only FV signing,
but also Hub4 signing, for example, or
software update signing.
I would say this is a quite convenient
tool if you have to do signing in the
embedded world.
Tried
Yes, and then if we have signed
everything, we put this in our own vendor
archive and go to the next stage where we
build a disk image with the LBE
Linux builder.
LBE takes an XML
description of how your Linux system
should look like.
For example, we describe the partitioning
there and we say what apt repositories
shall be connected. So here we connect the
upstream update archive of Trixie, for
example, and our own vendor archive.
And then we run LBE and it will install
all packages that we have defined in the
XML file and install it into a root file
system and also then create the
partitioning layout on the full disk
image.
The outcome of this is the disk image
itself containing the root file system,
but also DM Verity, root hash, this is
exported, and
uh the Linux UKI that we have previously
installed with the this uh
uh how was it called
Unified kernel package
then this is what LB can do out of the
box.
Now we are not fully finished, we have to
do some post-processing because we need to
turn this root hash into a UKI add-on.
This is not yet built into LB, maybe we
could get support for it.
Currently this is a a custom step uh
calling U um UKFI for the root hash
bin, turn it into a root hash UKI add-on
that has the root hash in the command
line. Uh we patch it back into the disk
image and get our final disk image this
way. And we also need something to do the
over the year updates.
So we extract from this final disk image
those partitions that are required
for uh doing a software update with SW
update.
This is what the partitioning scheme looks
like.
We have, as I told before, redundant
partitions for the root file system and
the verity partitions, but we do not have
a redundant API system partition.
This is a UEFI limitation. If we would
want to overcome this, I'm
not sure. We can at le we can for sure not
reuse system deboot then for the AB
selection. We could maybe trick UEFI into
doing this
AB selection by the next boot variables.
UEFI by itself supports next boot
variables where you could
implement such AB selection logic by your
own.
In this case, we do not do it.
We just use a single AFE partition because
it's good enough.
We have redundancy file-wise in there.
All what we need for a boot, the UKI and
the UKI add-ons are
side by side, A and B. in the file system.
So this sh should be good enough for
redundancy.
Then I said trust enrollment becomes
simpler.
What does this mean? If you're using UEFI
we have to to enroll the
following keys. W we have in UEFI the
trust DB, this is the main trust store
where loaded images are validated against.
Then we have the platform owner key.
This is we could think of it like a BOS
passport.
The one who is platform owner can modify
DB.
They cannot directly tell which images
should be bootable, but they can modify
the trust store. So this is some kind of
indirection.
Then we have yet another helper key.
This is the key exchange key. This is good
for platform owners to delegate trust for
the DP updates, to do it not by
themselves, but by the running OS.
And we have a revocation database, which
just will contain certificates that are no
no longer allowed. And this was for some
time the only way how UEFI could do
revocations. It turned out to be not
scalable.
And then they implemented a new technique
called Secure Boot Advanced Technology, I
think, quite generic term to do a real
counter-based
revocation per component, per boot
component.
I think this evolved around folks from
Microsoft and Red Hat.
Not sure who was Debian involved?
I'm not sure.
But Debian is using it.
Then
These are the keys that we need.
Now we want to bring them to the embedded
device.
How does this work? We first have to turn
our certificates into signature lists.
There is this search to FESIG list tool
that will just this is kind of format
conversion and then we we need to sign
this
certificate lists to get authenticated
variables.
And the signing can again be done with the
OpenSICA SIG Hub.
I have showed you before. It is capable of
signing UE variables.
And the result are three authentication
files.
What what to do to do now with them?
Formerly we had to enroll them somehow
with some hacks manually.
Now we can just place them during the LB
build on the ESP partition
into a well-known fo folder, the loader
keys auto folder, and systemd will pick
them up and enroll them into the UEFI DB.
And it will just use generic UEFI API for
this.
There is no special handling in systemd,
all goes through UEFI.
So that when we power on our embedded
device for the very first time, we will
see the the message that is shown below.
Enrolling secure boot keys from directory,
nada nada, boot finished, trust enrolled.
So we should do this step of course in a
trusted environment, not in the field.
We can do something similar for the TPM.
Also the TPM needs sealing and enrollment
of keys.
We can set up some system D machinery for
that.
And the basic idea is that check if the
data partition is already encrypted and if
not start the enrollment.
First encrypted with script setup, then
Create an explorer file system on it and
then use systemd crypt setup to enroll the
TPM keys and seal the new systemd provided
key slots to the
TPM. Yes, and from this from this point
on,
the encrypted disk can only be unlocked if
the TPM allows it.
We can bind the TPM to PCR registers or
not.
This depends on your use case.
PCRs are registers that Contain the state
of the system
by continuously measuring specific events
into it and
enhancing the
hash that is into it. And at a certain
point in boot time, we then compare, let's
say, the PCR 7 hash value, the actual
value, to some expected value.
If it matches, the TPM will unseal.
If not, no. But I can't give a general
recommendation about which PCR is
useful in your UK use case and if you
should use it at all.
So we are still in our trusted
environment.
the device a second time and we will get
this message um where we see that the
encryption is taking place, this will take
some minutes and then that the encryption
was successful and the TPM is sealed.
So TPM enrolling done.
Now we can go with our device into the
field and boot it.
And we have already seen in the first
picture that this is chain loading of
several bootloaders
where UEFI loads shim and shim loads
system deboot, system deboot loads
UKI which has a stub that loads Linux.
And all these load steps actually call
back to shim through a hook and
the hook calls back to UEFI so that every
load operation actually does the UEFI
Authentication against the UEFI trust
store.
Shim actually has some more ways to do
this.
There are protocols for shim aware
bootloaders
that could be called explicitly, but
systemd-boot is not using that, at least
not in the since version 258. They just
rely on
shim hooks, which for us is a bit
this this only works if um shim was built
with a vendor certificate.
Built in. Only then these hooks are
enabled.
And as I said, we don't really need this
vendor certificate.
So a naive implementation would build shim
without.
And then we don't have these hooks.
And then all this verification chain does
not what we suppose it to do.
So important thing here,
vendor certificate that needs to be
compiled even if we do not need it.
The original reason for shim, I just said
it, is that it could be signed
by Microsoft after public review on the
Shim Review GitHub issue page.
Vendor by vendor, and we could embed our
own certificate into it so that we can
then sign everything else with our own
signatures, with our own keys.
Maybe useful for you, not useful in the
case that we have explored uh up to
now. But what is very useful is this SBAT
verification feature of Shim.
And this works by having
SBAT variable in non-volatile RAM managed
by the UEFI, it's part of the UEFI Trust
Store, which contains for every
boot component
some expected generation number.
It's just really just an integer that's
counted upwards.
So we have, for example, SBAT should be at
level 1 and some other package should be
at level 4. That's what we persist at the
system.
And then the loaded binaries all have a
special PE section where exactly
the actual value is compiled in.
So we say here, this system deboot.
has generation number one.
And on boot, Shim will compare this, the
the expected value against the actual
value and deny boot if if it is if the
expected is lower
than the actual
yeah this is already known and used for
the boot loaders, but
my proposal here is to exactly reuse this
technology for the root file system root
hash.
then we co can assign a a rollback pro
counter to it and that's the whole idea
Yeah, this this is what I just said.
We can when when doing the UKFY of our UK
I add-on with the
root hash tell what SBAT generation number
it shall have.
And by the way, in the in the case of the
bootloaders, maybe this could be confused
that that generation number, this integer,
is not the same as the component
component version. The the generation
number is much more coarse.
It's only bumped and incremented after
some batch of severe C V Es was
fixed. So this happens maybe fourth a year
or so, or only twice a year for a
component and that they bump the
generation number.
Yeah. Okay, let's go further in the boot
chain.
Um now let's say we are at the stage where
Linux is about to load.
Uh so we will the systemd-boot will take
the UK I and will
load it and the UK I stop in here, which
is also
comes from the systemd project.
It searches the ESP for add-on files.
The add-ons are placed also in well-known
directories and it loads them one after
the other and merges their initial ROM
disk sections and the device tree
sections and the command line sections
into the main UKI.
And only then after this merge step the
Linux kernel will be booted with the
merged command line.
And I told you that we have formally
created a unified kernel
image with a Dracard generated init in it,
RAMFS.
And this is important because
we can give
These parameters to the Linux command
line, systemd verity root data equals
some some partition label and the same
also for the Deem Verity root hash
partition. And the rest will really out of
the box be handled by systemd.
It will the formally be set up by Draycut,
and then everything is built into system
D to to handle
creation of the block f the block device
and mounting with the correct root
hash.
This was before we had system D also a lot
of custom initial RAM file system
hacking. Now it's it works out of the box
Okay, we we have now booted it we have a
running system in the field then
maybe some words about
how much time do I have left?
Okay
This this just gives an overview how we
could generate the software update image
from the disk image. So we just extract
the parts that we really need from the
disk image. These are
the
the UK not the full ESP, but only the UKI
and the root file system of system
A and the Verity partition of system A.
And then we supply the root hash
separately.
And we maybe, if we want to bump this SPAT
policy that I told before, also
build another IFI binary that contains the
a new SPAT
policy. We have all it into a software
update file.
And
this
is not only data. The software update file
only contains a small program part where
we specify how this content should be
applied if we are right now on system A
or if we are now right now on system B.
Software update has to do different things
depending on whether we are on A or B and
this is defined in this receipts in the
SW description.
Yeah, and on installation, this is what
happens.
We upload this file to through some local
web UI or over the update
and
then
system D
software update first will check Are we on
A or R V on B
from that information selects the correct
slots to use in the software update file
and the correct logic and then copies the
artifacts from the SW update file to disk.
And then we do, at least if you are on
x86, a boot
control set default UKIB. This is telling
the boot manager that we are right
now on the B partition and that
This needs some blessing stage.
Blessing means we are not yet sure if this
will work.
We have to try it once, and only if the
boot works successful, then the new
B partition or the new B path is blessed,
as we say, and
this
loader setting will be permanent.
And the whole logic
is driven by systemd-boot. It depends on
UIFE variables and
file naming scheme, which is bad for the
case when we have U-Boot, because U-Boot
doesn't support writing UFE files, uh UFE
variables from the user space.
So we we need a a workaround if we are
using U-Boot.
Javi has some workaround for this.
I don't go into detail details now.
Yes, and this is
this would be about how do we now revoke
our root file system.
If we have discovered some program in the
root file system has a vulnerability, we
have to bump the
This integer counter, create a new UKI,
and we also have to tell
the existing system from now on the
expected value of this root file system
counter is two, no longer one.
And this is done by either installing a
new shim that has the new policy built in,
or there is also a special convention for
shim that we can supply a revocation
every
tiny binary that only contains the new
policy and nothing
else.
And this is how rejection looks like then.
If Shim detects we are trying to ru to
load a revocated root
file system, this we get this security
policy violation and the boot stops.
One word of caution here. System deboot is
not supporting WatchDoc.
And this is one of the reasons why FEBOOT
was invented.
Uh it ha FE Bootcode has built in native
watchdog drivers, systemd-boot not.
We don't need that on ARM sixty four
because there the watchdog ARM is done by
uh U-Boot and Linux kernel does the
watchdog handover.
FE Bootcod does nothing in that record,
but on X tai X eighty six is it's
different. So we have to fill this gap
somehow, at least for X eighty six.
And my proposal here is to factor out the
native uh watchdog drivers from FE
Bootcode and start it as separate FE
binary by systemd-boot.
This works, I have a proof of concept.
However, I would like it more if systemd
boot would implement this upstream.
And then if we are talking about upstream,
I have also done a systemd upstream pull
request that got an approved two days ago,
where we embed Dwarf systems
in the final AP file. And this means that
if we want to use
that at Debian, we have to enhance
DHStrip.
Is anybody is a developer of DHStrip here?
Because I would like to talk to them.
Yeah, DHStrip should learn how to strip PE
executables.
That's my
request for help to you.
Okay,
and that's it.
I am not sure. I think we have no more.
Okay.
We can talk
on the floor if you have questions.