Hello everybody. This is the last session
of this morning and being a physicist
myself I'm really happy to see an update
of Federico and Nikos for the controlling
CERN's accelerators with Debian.
Thanks a lot. Thank you.
So hello everyone. As Andreas said we will
be speaking about CERN.
I will disappoint you because it's not an
update.
It will be
repeating what we already said in Brest,
because as the Latin were saying, repeated
Juvent, repeating helps. Uh I'm Federico.
I'm a software engineer. I worked at CERN
since 2014.
I'm a free so free and open source
software enthusiast, however with little
free time to make uh concrete uh
contributions.
And I'm sorry I'm Italian, I will keep
moving and I hope to not fall from uh
uh so yeah, as well. So I'm Nikos Also
system engineer at CERN for about five
years now, open source maintainer for a
few things.
I've been using Debian for over ten years
now and I've been a maintainer of packages
for about seven years and hopefully a
developer at some point in the future.
So I will start introducing CERN.
CERN should be a well-known organization.
However, there is always someone that
doesn't know exactly what we do at CERN.
So first of all, CERN is an international
organization, mainly member states from
Europe. However, it has collaborations
that are all around the world,
Israel,
India, Pakistan. We have collaborations
with
Japan, USA, all around the world.
So our main goal is fundamental research,
in particular particle physics.
And uh sometimes uh some of our internal
tools they become very popular, uh like uh
the web. If you don't know, in the early
uh nineties uh the web was born at
CERN.
But actually uh doing the web is not our
main goal.
Our main goal is to operate uh the biggest
accelerator complex uh on on the planet
and then host uh experiments in this case
uh the most uh advanced or big experiments
that are uh located around the the the
biggest ring, the LHC.
Uh you can see here pictures of these uh
big experiments and you can judge yourself
the size by looking at the people uh
walking there.
So these are really huge uh systems uh to
make uh discoveries about uh things that
are more than invisible or uh subatom.
Then uh these systems are generating a lot
of data and this data is streamed uh to
our internal data centers uh where we
store just partially uh what gets uh
actually fetched from the sensors because
uh otherwise it would be too much.
So there is a triggering mechanism that
will filter out uh what we believe is not
uh relevant and then we will store
petadites, petabytes of data in our data
centers for uh future uh analysis.
However, uh I need to focus more on uh
what is our uh subject and
uh we will not be talking about the data
center.
The data center uses Red Hat Enterprise
Linux uh nine slash ten or the derivative
ALMA Linux nine slash ten. We will not be
speaking about uh the experiments because
they are uh managed in a completely
different way and includes not just CERN
but also collaborations uh with the
universities uh all around uh uh Europe
and and all the other uh contributors uh
around the planet uh and uh the choice of
an operating system uh may vary.
Sometimes there is still CentOS 7,
sometimes CentOStream, sometimes Red Hat
Enterprise Linux, sometimes Alma Linux,
sometimes uh Yocto uh systems and so on
and so forth.
So instead here we will be speaking about
uh the accelerator.
So the the computers and the operating
system that actually runs the accelerator
and makes the beams of particles uh moving
around.
So as I said before this is the biggest
accelerator complex on on the planet so it
starts with linear accelerators that you
can see here.
And then it's a chain of accelerators from
the smallest to the biggest one
accelerating at the speed of light and
then charging the particles with energy
and then eventually smash them inside the
the experiments to see what's inside.
So to give you some numbers, uh the
accelerator complex is distributed over
forty-three uh square kilometers.
Uh the biggest ring is the LHC, the
smaller l uh ring is the SPS
accelerator. And then to make a comparison
in the red uh rectangle you have the
Geneva International Airport. So you can
imagine traveling from one side to the
other of uh the accelerator, it could be
challenging with roundabouts, semaphores,
traffic jams, and then you need to make an
intervention on on the other side.
So most of it is underground. So we have
some uh infrastructure that is uh on the
surface like the linear accelerators and
the smallest accelerators, but then when
it starts to get big uh the accelerator is
underground.
Otherwise we would need a crossroads uh
for beam particles uh and cars uh to not
uh collide uh uh among themselves.
So uh at its deepest point uh it reaches
uh 100 uh meters underground
and uh being this a very uh special uh
system uh there is not a factory for
super big uh the biggest accelerator on
the planet.
So uh we have a lot of custom equipment
that is
most of the time designed at CERN and then
we uh reach for collaboration with the
industries to actually produce uh what we
need to operate uh this uh this
accelerator complex.
So uh in total we have thirty-six uh more
or less uh kilometers of beam pipes uh and
seventy thousand uh cables connecting our
equipment uh to uh our computers.
You can see the mess of cables uh on the
left uh side.
Eventually they all reach uh a computer so
that we can uh remotely operate them.
So today we have around 2024Presenter
installed. So this this is my person one
of my personal uh development systems.
Uh so the cabling usually is done better,
but in this case uh it's not uh that nice.
Uh the cards that are you see here very
often are designed at CERN, so this
electronic device, because they need to uh
to control very special equipment.
And uh these numbers uh they keep uh
growing because we always have uh new new
needs.
But CERN is not just made of hardware and
cables and pipes, it's also made of
people. And like this community, we have a
very heterogeneous population.
So we have software engineers, like we
are, system administrators, we have
mechanical engineers, chemical engineers,
we have
civil engineers to design the caves for
the tunnels, but also our own buildings.
So think about specialty and probably at
CERN we have it.
We need to do a bit of everything.
And of course, science physicists
eventually they need to analyze all this
data and tell us the meaning of life.
So uh being this uh uh very big very
heterogeneous uh set of uh competencies
and people and it's a big machine uh we
strive for stability.
So we need a very uh stable system uh and
uh it's uh sometimes challenging to keep
it uh stable.
So uh narrowing the focus and speaking
about uh Linux uh so we are using Linux uh
since uh two thousand eight. Before we
were using uh Linux OS or OS 9 uh more
real-time operating systems uh and uh
since we moved to Linux uh we always uh
used the uh real-time patches on top.
And uh uh at first
We started uh with our own uh Linux
distribution uh made in collaboration with
Fermilab in uh in uh in Chicago.
Uh this was based on Red Hat Enterprise
Linux five, then this evolved in
Scientific Linux six, then eventually we
uh thought that it was uh better to
contribute to an upstream uh project like
uh CentOS instead of rebuilding our own
because there was not n there was nothing
very special uh about it.
Uh and then uh we were more or less on an
autopilot uh with uh Red Hat Enterprise
Linux uh derivatives.
But as you know the life of CentOS 8 was
very short So this had a premature
end of life and this opened uh some
questions So we were asking what what do
we do next? Our plan was to use CentOS 8
We were on autopilot and now surprise
surprise what do we do? So we started
looking around well you know the distro
watch you can find uh thousands of uh
distributions uh so which one is uh the
best one of course any of these systems uh
can work even Emacs can be used uh
eventually as an operating system but uh
you want something that uh fits uh your
need and uh your uh particular context.
So for us it was important to uh judge
what is the level of control and slash
maintenance that we can uh afford in uh
operating and operating system uh
configuring an operating system and how
well it integrates in the current uh
infrastructure. So of course uh coming
from a Red Hat Enterprise Linux based
system uh we started saying okay, let's
start uh exploring the cheapest, let's
say, uh solution. So we started
investigating the Red Hat Enterprise
ecosystem. So CentOS uh eight was not an
option anymore, so we started looking into
uh CentOS Stream and as you know CentOS
Stream so the purpose of the project moved
a bit instead of being after Red Hat
Enterprise Linux it moved uh in between uh
Fedora and uh Red Hat. So it doesn't
guarantee the the s same stability at Red
Hat Enterprise Linux uh but it's not uh uh
cutting edge as Fedora.
So our plan was okay, we try with CentOS
Stream 9 with the idea of upgrading to
CentOStream 10 and so on and so forth.
However, we knew that we could not afford
to have more surprises.
So the first thing we did is to make our
infrastructure more portable to other
distributions and having Debian as a Plan
B solution.
So I think I will not spoil anything if
eventually we are using Debian.
It was not the Plan B because Uh there was
something that uh eventually uh broke uh
the cameras back. And uh uh and what was
it?
Uh it's a simple compiler flag.
So this compiler flag M Arch optimizes uh
the code for the microarchitecture uh
Intel 64-bit microarchitecture version two
and this is applied to the entire Red App
Enterprise Linux 9 uh ecosystem.
And then on the version 10 you get the
optimization level uh increased to uh V3.
So this has uh consequences. Uh on my
person I can change my personal PC but if
you are speaking about uh uh a big
infrastructure like uh the the CERN one uh
this has uh big consequences. And uh for
us the consequences are uh depicted here.
So uh at the time uh I made the risk
analysis 2024 the first ALF
47% of our systems were Intel 64-bit
baseline, so CPUs like the Intel Core
Duo, meaning that the optimization at the
V2
was preventing the system to boot.
You get a message from GLib C, I cannot go
further than telling you that I cannot go
further. And then we had another 17% of
machines that cannot boot with an
optimization
level to x86-64-v3.
making a grand total of sixty five percent
more or less of systems that could not
work and not with the the operating system
proposed nor with the next uh update.
But this is not the only consequence uh
because uh PCs uh pre twenty ten were
generous uh in terms of uh PCI uh slots
and uh today they are not.
Uh you are like if you get one on your
desktop uh in the industrial systems uh
you might still get one or two slots if
you pay a bit more, uh but not enough.
This means that some systems that must be
reconverted, uh one system might need
three computers. And this occupies rack
space more than it was occupying before.
Then another problem we had is the VME
single board computer.
This is our workhorse for the control
system.
And at the time we were very exposed.
We are designing and now we got the new
generation, but you know, the production
might be slower because of our war
somewhere, because of AI somewhere else,
and the price increases and so on and so
forth.
So now
looking back at what I said before, this
means that we need to Upgrade 65%
of our uh systems. Some of our caves are
underground and uh it's a forty uh
forty-three kilometers uh uh square meter
distributed.
So you take your car, you load it, or your
truck, you load it with the computers,
you go, you go underground, you change a
U-Boot and perhaps you swap two cables and
you will discover this only later at
commissioning time, and this might happen
more often. Then you need uh more rack
space and perhaps in some places you don't
have rack space, so you start uh digging
more space to add your uh computer uh
there and uh find space for the cable
because sometimes as you see in the
picture there, uh it's full of cables, so
the shelves are not enough and some cables
are starting to be just laid on on the
floor.
And if you we add more because we have
more systems because we don't have PCI
slots uh and so on, it starts creating a
mess.
And meanwhile All these people must be
kept happy uh because otherwise they
cannot do their job. Uh so it's uh very
challenging.
So we estimated that this cost was more or
less five point four million uh Swiss
francs to upgrade uh the the system.
Uh eleven boards uh must be redesigned
because they were in PCI uh format uh
or uh anyway, the as a consequence uh they
had to be redesigned uh implying that you
need uh more electronic engineers for a
job that you didn't foresee, more soft
engineer to support uh the development of
driver and low-level libraries for the new
hardware coming, and more technicians to
go and do the installation on on field.
Reorganize the rackspace, reorganize the
cable, be sure that everything is correct
and the commissioning will be uh affected
uh eventually.
We estimated that with a bug-free solution
uh we had uh twenty percent uh success
rate. Uh this was very low. So uh we said
okay we have a software problem uh and we
will look for a software solution.
So and uh For us the software solution was
uh Debian, uh a universal operating
system that tries to work on as many
hardware uh as poss as as possible.
So uh the optimization level was not uh
yet introduced.
I know that today is uh under discussion
if uh it worth uh optimizing uh the the
compilation or not.
So this was also the chance to review our
uh internal uh architecture surrounding
the operating system.
So we started by decomposing the
distribution.
So we are using Debian for the initrom FS
and uh the user land.
However, we don't uh pick up uh the Linux
kernel from Debian, we build it from
kernel.org. We apply real time patches uh
and uh following some embedded uh system
uh practices uh we built in all the
drivers that we need the from uh from the
distribution and then grab as uh as a
bootloader.
This uh to allow us to be more flexible in
changing uh just one of these components
uh at any time. So we can change to Red
Hat Enterprise Linux if we want without
affecting the Linux kernel uh not just the
building process but so also the delivery
process and the entire system.
So then uh we renewed also the the the
delivery of our components uh we are uh
based on uh more cloud based with
Kubernetes.
Uh we delivered the the the GRUB over the
FTP and all the other components uh to uh
with the HTTP. It's a disk lex diskless uh
system uh and we have a configuration uh
service that tells us uh which system
needs uh which exact version of uh the
operating system, which kernel version,
which Debian distribution version.
All these artifacts are stored in a GitLab
uh package registry and whenever there
are changes uh they will be applied to the
system.
But now Nikos will go more in details
about uh uh what we actually did to
implement uh uh this uh this migration.
So we were at the point that we knew that
Plan A of continuing with Red Hat did not
work and we're going to go with Plan B
which is Debian.
But now we have the entire task of finding
out how to migrate a fully Red Hat based
infrastructure to a Debian infrastructure
while also implementing all of the support
systems in place to keep it up to date and
build it correctly and so on.
Uh the first thing that we actually looked
at is the release schedule.
Uh f because when the accelerator is
running it's running twenty fours a day,
seven days a week and uh we cannot do any
updates during that time.
And while we do have short technical stops
a couple of times a year, during those
technical stops uh we have so far not been
able to do major distribution upgrades
because major distribution upgrades can
affect many parts of the operating system
which then can affect drivers or user
space applications which then all need to
be retested and this is this has not uh we
did not have the time to do that in the
short technical stop. So you might uh have
a five-year plan.
At CERN we have a minimum a fifteen year
plan.
So we know the state of the machine all
the way up to twenty forty one.
Uh so uh up until now we've been using
CentOS 7 in this run.
So in green you can see when the machines
are running.
In blue you have the short term technical
stops for quick maintenance.
And in red is what we call the long set
down which is when we perform the big
maintenances replacing hardware, major
software updates and so on.
So up until now, up until the end of 2026,
we've been running CentOS 7 because we
did not have the opportunity to update.
And we are looking for to upgrade our
systems for the next run of the
accelerator, so deploy in the beginning of
27, and keep it as stable as possible up
until the end of 2023, when the
accelerator will shut down for another
long maintenance period. So the first
thing is we did is we know that Debian
stable supported for four years, we have a
four-year run cycle, so if everything
aligns, it should work, right?
Well, plotting the releases, we see, okay,
so we can start developing a bookworm,
then we can migrate to Trixie during the
long shutdown.
Trixie will go into end of life before
even the accelerator restarts.
Then we can update to 4K and 4K will go
end of life just a couple of months after
the commissioning ends. Uh so then we
would have to update to whatever Debian 15
is. So the stable releases were a bit not
too stable for our uh schedule.
So thankfully we saw that it is the
long-term support releases so we see that
okay now Trixie goes end of life just
after the commissioning but 4K in this
time stays until about one and a half year
before the run.
But plotting Debian 15 here we can we can
assume that it should stay supported up
until at least until the end of 2023.
Uh however we also noticed that uh there
is Friction that offers the extended long
term support releases which uh have a
really long support period and With those,
even if we've stayed on Bookworm for the
entire time, it is still supported up to
almost until the end of 2023. And Trixie
can definitely be supported from before
the run to even after the run up to
2024Presenter and SSH and 2024Presenter
and Real-Time 2024Presenter us because if
we find out that we cannot do major
updates during the run, this is a very
viable backup plan so we can stay with the
same distribution for the entire period.
So we devised the following uh the
following plan.
So our plan A is well develop and validate
everything on Bookworm and then during uh
uh uh near the end of 2026 we deploy
Trixie and we use Trixie up to
2030. Then we upgrade directly to Debian
50 2030 which should stay supported until
the end of our run. If we find out that we
cannot uh we cannot do major upgrades
during the run then we develop on Bookworm
uh deploy Trixie During the maintenance
period in 2026-2027, and then we stay with
the Trixie extended long-term support
until the end of the run. And uh the very
last resort plan is if we find out that
after validating a book home there were
blockers with Trixie that we could not
upgrade, we would stay for Bookworm with
the entire run.
However, uh by now this is one of the
updates.
We have now validated Trixie on all of our
systems, so we'll we will be deploying
Trixie at the end of this year.
Uh so as you see it's uh it's important to
say that the Debian LTS and maybe the
ELTS process uh uh the ELTS uh programs
were very useful for us.
So we have started sponsoring uh Friction
in order to try to keep the LTS and ELTS
releases supported and and keep them
coming as much as possible.
So thank you for providing them.
And uh going back to how we uh going back
to the technical stuff, okay, now we know
that we have a release schedule that can
work, but now we have the technical side
on how do we actually build and deploy
Debian on the infrastructure.
So going back to the selection side, we
see that we're going from a distribution
that has a very high level of integration
and expertise in the s in the cern
community to uh to a distribution that has
a very low level of integration.
So we do not have uh the staff does not
have as much knowledge about Debian.
Our infrastructure is entirely based on
RPM based systems.
So Debian does not integrate well in this
context.
So we were sitting at the meeting trying
to see okay how many people here have
experience with Debian. Some did, some
used it as their in their personal
machines or contributed, but most people
in general did not.
So we were looking along like okay does
anyone here know about this Debian thing,
about how it is and how it works?
Uh so what we did is we went back to
Friction uh like uh so can you help us?
Can you show us how to build a production
ready uh Debian infrastructure in order to
bootstrap uh the Debian systems at CERN?
So they sent us Seb, who did a
four-day-long Debian training, including
everything from the Debian philosophy,
package building in ve with various
methods, how to host uh apt repositories,
built repositories, and so on.
So thanks a lot for that.
And uh okay. And then after we did this
training, it was trying to review our
infrastructure about how we deploy it and
how we Uh and how we manage it.
So, a quick overview of our
infrastructure.
So, in the control system, our
infrastructure is split into three layers.
So, the first layer is what we call
consoles.
They are basically desktop computers in
various rooms about CERN.
Uh in generally, the biggest uh the the
biggest concentration is in the control
room. So, this is where your physicists or
operators sit and they have applications
written either in Java or or Python.
And they and these applications can be
used to send commands down to the
accelerator. So, and For example, it can
be oh, I need uh ramp up the magnet
strength to this or increase the partic
the the intensity of the beam to that.
So these are desktop applications that are
currently running Alma 10 with xfc
desktop environment and they will stay as
it is.
So this is our first layer. Second layer,
so after the operator sends a command,
they are sent to the data center where the
these commands are logged, authenticated,
analyzed to make sure that it is within
the safety parameters of the system.
Uh these servers are uh were sent to sent
to seven, now RHEL nine and will probably
move to Alma ten. So and then after this
command has been analyzed, verified, it is
then forwarded to the third layer which is
the system 100 meter underground.
They are paradoxically called the front
end computers.
However, if you're an electronic engineer
the computer that you plug your magnet
into is the front end and everything else
is the back end.
Uh so these front end computers uh they
are this is what will be migrated to
Debian so they are responsible for
receiving the commands from the data
center and then implementing those
commands with hardware signals.
So you receive a command okay ramp the
magnet strength to this value.
This means they will set the command to a
power supply to ramp the ampere adds to
this value that will increase the magnet
strength.
So they're basically the interface between
the electronics world and the software
world.
Uh having to do that, this means they have
a f a bit of a peculiar configuration, as
Federico mentioned. So, first of all, they
do not have any storage.
They are completely disclosed.
Every time they boot, they boot directly
from the network.
And second of all, since they have to be
plugged into very specialized electronics,
it means they have custom PCI, VME, and
other such hardware cards, these are PCBs
designed at CERN for CERN. Uh they are
well, technically they are public, but I'm
not sure someone else uh would have a use
for them other than if they want to make a
particle accelerator. Uh but
Because we have custom hardware, it means
we also need to have custom kernel
drivers. And maintaining the stability
when you have custom return kernel drivers
is uh much more difficult than if you just
had user space applications, because now
any bug in a kernel driver can affect the
stability of the system.
Uh so how d how did the infrastructure
work in previously?
So previously we had uh a single bare
metal hardware server that was the boot
server. This served the GRUB and kernel
boot configs over TFTP and then the NFS
root directory was also on the same
server, it was served over NFS and Kerner
and the kernel did an uh NFS mount of the
directory, so it was read-only.
Uh this has some advantages like that
every system we know has the same root
directory. It also has some big
disadvantages specifically that now this
server is the holy grail of the data
center.
We do not approach this server, you do not
breathe of the on this server.
Because if this server goes down a lot of
people at CERN will have a very long day
because it means all the two thousand
embedded systems suddenly lost their root
drive. And for application delivery it is
a similar system.
There is another single NFS server that
has all your applications, shared
libraries and kernel objects and it serves
them over NFS so that they can then be
run on the systems. And our deployment
mechanism is similarly you build your
application and then you S CP it to the
server at the correct location.
So
This uh this architecture was designed
back even before 2005 for the first run of
the LHC. Uh it was kept simple on purpose
to try to have as l uh the least moving
parts as possible. However, now we're in
2026, we have new technologies.
We can for sure do something better.
Uh first thing that we looked at is how do
we actually build Debian and how do we
make a Debian image and how do we actually
develop software?
Because remember most of our
infrastructure is staying on uh RHEL 9 or
Alma 10. Uh so we need to be able to
develop software, compile software that
will be run on Debian, however on Alma or
Red Hat.
So this we looked at many different
solutions for building Debian images.
LBE stood out specifically because one of
the outputs is not only the built-ins, but
an SDK cell script. So this is a cell
script that if you run it, it turns your
your Linux distribution from where
whatever it is into the Debian image that
you built. So you can be in a Red Hat
system, you run the script, and now your
entire environment, the GCC and all the
other built-ins you need, are coming from
the Debian image. So you can basically
compile Debian software on Red Hat.
And uh how it works is basically you have
a manifest XML that describes all your
packages, you have an overlay directory
for any files you need to override.
You feed that into LBE, LBE spins up a VM
and out comes your Debian image and your
SDK.
Uh additionally, there is this new big
thing coming about, which is Kubernetes.
And we we had the challenge of okay, can
we have a boot server on Kubernetes?
Currently, we have this big bare metal
server we cannot touch.
Can we try to distribute it onto multiple
servers and since uh the way of the
industry is going to Kubernetes, can we
run this server on Kubernetes?
So uh we actually hit uh one roadblock
quite early which is TFTP.
If you know a bit about TFTP, it's a Unip
UDP based protocol and the session
handling is a bit weird, so you Ask the
TFTP server on the default TFTP port,
please send me this file, and the TFTP
will report back on a totally randomly
chosen port that does not follow the
normal Cession semantics.
So having TFTP go through any sort of NAT
or firewall is very difficult.
So at this point here, we just said okay,
let's do the simplest thing, which is you
install TFTP on all your Kubernetes
ingress nodes.
So they all serve the same files, which is
basically the GRAB config.
And we say, okay, so the first layer we
just install TFTP on all nodes, all nodes
serve the same thing. And after GRUB is
loaded, GRAB thankfully supports HTTP, so
we can be in more known ground in the
Kubernetes land.
So after Grab has booted then we need our
kernel, the d the initramfs and finally
our distribution. So for that it is more
simple because it's all HTTP based.
So we have an so uh however another thing
is to note is that we want to be able in
control with all the infrastructure to
have different versions of l of the kernel
in RAMFS of the distribution in case in
case we need to for example oh there is a
bug on this uh on this uh type of system
for on this version of the distribution.
We do not want to touch the entire of two
thousand systems and upgrade all of them.
Again, we need stability. So we want to be
able to say okay this computer gets
updated, all the others stay the same.
To do that basically we have so we have
our GitLab registry that Has all of the
artifacts. So when we built a kernel, it's
on GitLab registry.
When we have a distribution, it's also in
the GitLab registry and so on.
And then we have a database that contains
all the information of which computer
boots which version of kernel, which
version of the operating system, and so
on. And we, through a script, we basically
fetch the configuration and then download
all of the artifacts into a static
directory tree and then serve them over
NGINX. So this means that the entire the
entirety of the boot server is immutable.
And the one missing piece here is okay, so
we have the static serving of the files.
We can boot. What happens if we need to
update?
So this is also a Kubernetes operator
written in Bus that just fetches the the
configuration from the database every so
often and compares it with what is being
served by the boot server. If ever uh if
anything has changed it simply sends a
redeploy command to the main boot server,
we just deletes and restarts the
container. So the entire boot server is
stateless, you can destroy it and recreate
it at any time without having to keep any
state.
So operating system is here, but an
operating system does not do much if you
don't have any applications. Uh the old
way of deploying applications as I
mentioned is uh through a s uh shared NFS
server and for kernel obs uh kernel
objects specifically kernel objects have
to be compiled for a specific kernel
built-in and this means we had multiple
directories in our NFS server.
So okay so for kernel 3.10x we have one
set of kernel objects, for 4.14 we have
another set and so on.
Uh how would we translate it to Debian?
So the Orthodox Debian way of doing that
is you have your repositories, you have
your application in the repository, your
rubber in the repository, and your drivers
also in the same repository. However, when
you install the driver, because it has to
be customized for the kernel version, then
you do not download the kernel object,
you download the source code and then you
use DKMS to compile it.
However, uh our systems, as we mentioned,
they are embedded systems, they have very
limited RAM, they have very limited
resources, and uh for sure not enough to
run a full DKMS built-in. So this option
was not uh was not suitable for us.
Uh we could s we're like okay, can we do
the same thing but somehow pre-build the
kernel objects inside the Debian package?
We could, but then this means that we need
uh different binary Debian packets per
kernel version. This means we need to have
a source packet named driver one, and
then this driver one would have uh binary
packages named driver one dash five ten
dot two two five dash RT one one seven
dash one, which does not roll off the
tongue. And additionally, having looked at
it, uh making uh deb helper make dynamic
binary packages based on the source
packets is not really something that can
be done. Trust me, we looked.
So what's the alternative? Is the
alternative we split a repository instead?
So we still have our main repository, it
has the application debs, our library
debs, but then this uh this application
will depend on driver on driver one and
then driver one can come from a different
repository depending on the kernel version
that the machine is running. So for
example here we have two repositories, two
driver builds, while they are the same
packets, they are different builds for the
specific kernel version. And basically
during the boot sequence the first thing
the system does is checks okay what kernel
version have I booted and then it
configures the correct repository for that
kernel version.
So now we have the management of okay we
need to every time we deploy a new kernel
version we have to make sure to rebuild
all the drivers and to create a new
repository. to host all those drivers.
So uh this is our current system, this is
how it works.
What have we learned from this entire
experience?
Uh first of all just appreciations to the
Debian community-the reasons we switched
to Debian in the first place is because
our systems were not no longer supported
on Red Hat. So thank you to Debian for
keeping it a universal operating system as
the subtitle is. And also again to the
Debian community that has managed to make
an operating system that is stable enough,
usable enough, that can basically rival
that of Red Hat and at some point even
exceed Red Hat as in this case.
Uh though not everything is uh amazing, is
we had a few challenges on the way.
So the first thing that we hit is that
there is no standard tooling for
automating building and publishing on f
packages.
Uh of course yeah you have built-in, you
have DAC, but these are very specialized
to Debian and I think people can know that
if you want to run uh DAC on your systems
it is not uh easy to set up or easy to
use.
Also there's no official upstream policy
on how to deal with binary kernel modules
so we basically just had to make our own
policy on the way and see what works, what
doesn't work. And uh Debian as a package
building system looks very promising.
Uh we're looking forward to have it uh
support more features.
However sadly it has some blockers that at
the at this point it we cannot support in
our infrastructure. Uh another difficulty
is that a lot of tools do not support uh
having multiple versions of the same
package in a Debian repository.
This is something we want, as I mentioned
because when we update something we want
to update the package on one system and
not on the others.
And since these are live systems, if a
system reboots it needs to be able to
download an old version of the packets.
So this makes a bit uh life difficult on
how we build uh Debian repositories.
Specifically I know there is Repropo,
we're using RepRepo.
However the branch that supports uh
multiple package versions is still in
experimental as of two days ago, I
checked.
Uh so either we have to have multiple
repositories for each revision of the
system or we need to move to something
like uh uh FTP archive or other tools.
Uh for building we evaluated Debian and
OBS and other options for building Debian
packages. However, all of them had
limitations and as I mentioned we have a
very extensive RPM built-in infrastructure
at CERN based on CODI.
So Kodzi if you don't know is the RPM
built system that is used by Upstream
Fedora. So it's like okay so Debian is not
currently the biggest part of the
infrastructure and we already have a huge
infrastructure in place for RPMs.
So instead of bring instead of bringing
the Debian infrastructure in, why don't we
bring Debian to the RPM infrastructure?
So I want to give a a quick shout out to
the technical to our technical student
Eric. So he basically developed a plug-in
to add the Debian packets building support
to CODSI. So now the Code Z D RPM built
system can also do Debian packages.
Uh final takeaways is do not assume that
if you have a working system on one
version of the distribution, your system
will keep working if you update the
distribution to the next major version.
This is exactly the reason that we're
here.
Being able to not being bound to a
specific distribution and having the
ability to switch if and if any of that
happens is a very good thing to keep in
your infrastructure. So currently we have
validated Trixie for our infrastructure.
We have a couple of dozen machines that
are operating the accelerator.
However, uh the accelerator is shutting
down in two days, first of September, all
of the machines are being shut down.
And by the end of the year, uh basically
all two thousand two hundred of the
embedded industrial computers will be
migrated to Debian Trixie.
Thanks a lot for this entertaining and
interesting talk.
I have a first question, because I
remember I was invited to a Debian Syme
Sprint in twenty twelve in ESRF in
Grenoble.
And what do you think about the next
MiniDebConf Switzerland at CERN?
And we do a sprint together to Develop
something.
I think it's you could talk with the
people, but I think this could be really
nice. Next year. Next year. We can discuss
bureaucracy will be on the way.
Yeah, yeah, yeah, sure. But probably, but
I remember that in Grenoble it was finally
they initialized this and it came from
there and you are even bigger than
Grenoble, right? So this should be no
problem at all.
Definitely it's an option to have it at
certain infrastructure to do it.
Yeah.
To go through the public administration
bureaucracy.
But we will come back with news.
Okay, this would be great. Other
questions?
Did you ever consider the option of
rebuilding CentOS with reduced compiler
flags? And if so, why did you reject that
option?
Yes, it was on the table, this option.
We also discussed this exact option with
the Alma Linux team.
However, no one wanted to become
responsible for it because, anyway, it's
not that you remove the flag and it's all
fine.
you need anyway to go through a validation
process.
And our main goal at CERN is doing
particle physics.
So the people we are not there to make
Linux distributions and that's why the
first time we got rid of scientific Linux
to rely on communities or
companies for this component. So
insourcing this responsibility means that
you need to also provide resources to keep
maintaining it.
And the same we got the same answer from
the Alma Linux team that at the time when
we discussed the team was just burning and
they didn't have enough resources to
commit and saying yes, we guarantee you
that the baseline will work fine.
So they didn't want to commit at the time.
But this also means that you trust us more
than you trust yourself.
That's it's a huge compliment.
Yes. Thank you very much.
So it seems you care a lot about having
your front-end terrors, as you call them,
very up-to-date. Do you have obviously
they are not exposed on the public
internet, right? So is there some security
concerns that you need to address?
Can you explain to us w what type of
things you want to address there?
And why you think you must upgrade and
it's not okay to run a vintage Debian
version? So they are not on the public
internet and they're even not in the CERN
network. So they're what in what we call
the technical network.
With CERN has two networks, so the main
network you have all your desktop
computers and so on. But the technical
network is not even it's not just a VLAN,
it's a physical separated Switzer
infrastructure.
So they're in an entirely different
network domain.
And Give it's more secure of course
because you have to go through two layers
to pass it. But also there is still it is
still good to keep the patch level up to
date. We do not keep it of course at the
same level as the bleeding edge.
We usually only update uh uh do software
like security updates once a year during
the short technical stop. So we do major
upgrades during the long shutdowns mostly
and every technical stop at the end of the
year we do patch updates.
Uh however it is still important to keep
up to date as much as possible because for
example if we kept running with uh Debian
twelve and now uh at the end of the run
Debian sixteen has been released, go
porting our software from Debian twelve to
Debian sixteen is a lot harder than if we
do incremental updates.
More often. Well, about update, if I can
add, this is the same situation we ended
up with CentOS 7. So to until few months
ago, CentOS 7 was the solution.
If you want to deliver something, you need
to use GCC version 4.
You cannot use the latest C standards.
So the environment is preventing you from
using more modern technology.
So you are in an environment that is
frozen in the past.
And being a release cycle very long.
So it means that
what we do is obsolete and also in terms
of people coming and going and students,
they will come and you cannot use C ⁇ 20
because here we use only GCC 4.
And some people started maintaining their
own compiler to be able to move on.
Some people started to get other libraries
because they needed that specific feature
that appeared in version, I don't know
what.
So anyway, it's good to keep the
environment
Modern, not cutting edge perhaps, but
still having a regular upgrade policy is a
good policy. Do you plan for rolling
updates for testing? So I come from the
industrial field, so when we have
customers, they
doing rolling distribution, testing that,
but don't roll out for exactly the same
reason. But if they face this problems, so
they reach them early, they do
all the integration work. And if they have
a security breach or something like that,
so they can always see of if they backport
or going forward because
if it's major enough to do that and they
have the test coverage Almost.
So uh
up until now, all the validation was done
sort of manually.
So we validated the hardware catalog uh
manually against uh Debian twelve and uh
Debian thirteen with the kernel five ten
and now six twelve.
However, right now we are working on uh
and a project to actually automate all
this process, to have our development
branch, for example, on the kernel and
whenever we make a commit, the idea is
that it triggers a test on a number of
machines that are pre-configured to run
tests of that type so that we can uh uh
reassure our our internal clients Uh that
our the system that we are providing is uh
booting and working at least with the
hardware and the basic setup.
Then, of course, corner cases we cannot
test uh everything, but uh we try to have
uh 80-90% coverage being automated.
And then uh you can you have to also to
imagine that some tests cannot be
performed without the beam being active.
So because that hardware works only if
there is a real beam uh or the real
equipment that is very expensive and and
big and you cannot have it in your lab.
So for this those are perhaps a more uh
niche test and those they will be kept
done manually.
But yes, we are doing it uh now.
Thank you.
Any more questions?
This does not seem to be the case.
Oh.
Reproducible. Um
The experimental branch that you talked
about that still needs more testing and I
would like to have your input if you can
switch to that branch.
Because I actually uploaded it to Unstable
once and then I got like
a flood of emails coming in, oh, my Rep
Repro broke and that's why it's not in
Unstable yet. Thank you for saying so.
This is currently our our plan, so we want
to download that branch and test it.
So we will get back to you.
Last chance for questions?
If not, thanks a lot. And again, some
applause.
Thank you.