Intro
As Sys Admins, we have all had to go through it. The moment when you type reboot and hit enter on a production server. You wait what seems like forever, holding your breath. Then you open up a terminal, and start to ping the machine with no reply. It starts to feel like all the blood in your body wants to move into your head. You anxiously log into the power strip, and just before you reset the port…ICMP response.
How can we make this better? Patch the kernel in memory.
Years ago, ksplice was all the rage. A project called uptrack started writing the patches and releasing them for everyone to use. Then Oracle acquired it, and stopped sharing. The ksplice function is still available, so another project called kpatch arose from the ashes.
My understanding is, Red Hat will push these to the repo as kpatch-patch-$kernel_version. I’m not sure how that will translate to CentOS 8, or if it will at all. In the event that we are left high and dry with no kpatches, heres how you roll your own changes.
Dependencies
I am working on a fresh CentOS 8 install in AWS. So, to get started, lets go ahead and install some of our dependencies.
dnf install rpm-build git gcc elfutils-libelf-devel bc bison flex kernel-devel openssl-devel make python3-devel -y
kernel source
We need the kernel source, and a few other things. Under normal circumstancs, you should be able to type.
dnf --enablerepo=base-debuginfo install kernel-tools-debuginfo kernel-debuginfo-common kernel-debuginfo kernel-debug-debuginfo -y
note: see the below workaround if dnf does not find the packages.
Work around
At the time of writing this, the CentOS8 debuginfo repo doesnt have the metadata generated properly. So we need to grab these manually from a mirror… uname -a, and make sure you get the one that matches your running kernel. In my case, I am running 4.18.0-80.7.1.el8_0.x86_64. So those are the versions I will get.
dnf -y install http://mirror.facebook.net/centos-debuginfo/8/x86_64/Packages/kernel-debug-debuginfo-4.18.0-80.7.1.el8_0.x86_64.rpm http://mirror.facebook.net/centos-debuginfo/8/x86_64/Packages/kernel-debuginfo-4.18.0-80.7.1.el8_0.x86_64.rpm http://mirror.facebook.net/centos-debuginfo/8/x86_64/Packages/kernel-debuginfo-common-x86_64-4.18.0-80.7.1.el8_0.x86_64.rpm http://mirror.facebook.net/centos-debuginfo/8/x86_64/Packages/kernel-tools-debuginfo-4.18.0-80.7.1.el8_0.x86_64.rpm
Install kpatch
[root@ip-172-16-10-43 ~]# git clone https://github.com/dynup/kpatch
Cloning into 'kpatch'...
remote: Enumerating objects: 47, done.
remote: Counting objects: 100% (47/47), done.
remote: Compressing objects: 100% (42/42), done.
remote: Total 7136 (delta 25), reused 17 (delta 5), pack-reused 7089
Receiving objects: 100% (7136/7136), 2.01 MiB | 27.39 MiB/s, done.
Resolving deltas: 100% (4291/4291), done.
[root@ip-172-16-10-43 ~]# cd kpatch/
[root@ip-172-16-10-43 kpatch]# make
make -C kpatch-build
[...]
make[1]: Leaving directory '/root/kpatch/contrib'
[root@ip-172-16-10-43 kpatch]# make install
make -C kpatch-build install
make[1]: Entering directory '/root/kpatch/kpatch-build'
/usr/bin/install -d /usr/local/libexec/kpatch
[...]
make[1]: Leaving directory '/root/kpatch/contrib'
[root@ip-172-16-10-43 kpatch]#
Nothing fancy here. We just cloned the kpatch github repo. Then we cd in, and make && make install to build it.
Make some changes
Now, we should have a machine with all of our dependencies installed, and the linux kernel source. Lets navigate there.
[root@ip-172-16-10-43 ~]# cd /usr/src/debug/
[root@ip-172-16-10-43 debug]# ls -l
total 0
drwxr-xr-x. 4 root root 171 Oct 1 22:23 kernel-4.18.0-80.7.1.el8_0
[root@ip-172-16-10-43 debug]# cd kernel-4.18.0-80.7.1.el8_0/
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# ls -l
total 2680
drwxr-xr-x. 20 root root 236 Oct 1 19:03 linux-4.18.0-80.7.1.el8_0.x86_64
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]#
Your version may be different depending on whats current when you read this. Lets copy the kernel directory, and name it something obvious, like kpatch.
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# cp -R linux-4.18.0-80.7.1.el8_0.x86_64 kpatch
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]#
Once we make our changes to the kernel in the kpatch directory, we are going to use the original to make a patch(its actually just a diff).
Building the kernel patch
Changes
This can be anything. It can be updating your kernel from one version to another, or it can be just adding custom changes. With CentOS8 having just come out, there is no other kernel version for me to move between. Instead, we will make a custom change.
note: If you want to do this with a newer kernel version, just grab the debuginfo packages(which contains the source). Then create a patch(diff in a text file) between the version you are running, and the version you want to be running. Then follow the rest of the steps using that diff instead of mine.
Here are my changes.
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# vim kpatch/fs/proc/meminfo.c
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# diff -bur linux-4.18.0-80.7.1.el8_0.x86_64 kpatch
diff -bur linux-4.18.0-80.7.1.el8_0.x86_64/fs/proc/meminfo.c kpatch/fs/proc/meminfo.c
--- linux-4.18.0-80.7.1.el8_0.x86_64/fs/proc/meminfo.c 2019-06-24 10:14:47.000000000 +0000
+++ kpatch/fs/proc/meminfo.c 2019-10-02 14:52:14.344644314 +0000
@@ -38,6 +38,7 @@
long available;
unsigned long pages[NR_LRU_LISTS];
int lru;
+ long teehee = 10 ;
si_meminfo(&i);
si_swapinfo(&i);
@@ -52,8 +53,8 @@
pages[lru] = global_node_page_state(NR_LRU_BASE + lru);
available = si_mem_available();
-
show_val_kb(m, "MemTotal: ", i.totalram);
+ show_val_kb(m, "TEEHEE: ", teehee);
show_val_kb(m, "MemFree: ", i.freeram);
show_val_kb(m, "MemAvailable: ", available);
show_val_kb(m, "Buffers: ", i.bufferram);
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# diff -bur linux-4.18.0-80.7.1.el8_0.x86_64 kpatch > test.patch
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]#
note: don’t mess up the order in your diff. Its original first, then modified second. If you do this backwards, it will tell you the patch was already applied and fail.
Here, we added an arbitrary value to /proc/meminfo. We should see a new directive called “TEEHEE”, and whatever that long teehee = 10 converts back to when printed. But it will give us a litmus test to confirm that our changes are actually in the running kernel. Note, the second time we ran diff we output it to a file. We will use this to actually build the module.
Building
This part is going to take a little while. So you should probably get this started, and then maybe get some tea or something. I made the mistake of using a dual core VM and it took about an hour to run the first time.
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# kpatch-build test.patch
Using cache at /root/.kpatch/src
Testing patch file(s)
Reading special section data
Building original source
Building patched source
Extracting new and modified ELF sections
meminfo.o: changed function: meminfo_proc_show
Patched objects: vmlinux
Building patch module: livepatch-test.ko
SUCCESS
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]#
You can actually monitor the progress by opening another shell, and tail -f ~/.kpatch/build.log. This is also where you check if it fails to build. So far the cause has mostly been devel packages I forgot to install, and my shitty C.
Test
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# kpatch load livepatch-test.ko
loading patch module: livepatch-test.ko
waiting (up to 15 seconds) for patch transition to complete...
transition complete (2 seconds)
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]# cat /proc/meminfo | head -n 3
MemTotal: 7697340 kB
TEEHEE: 40 kB
MemFree: 1275160 kB
[root@ip-172-16-10-43 kernel-4.18.0-80.7.1.el8_0]#
If you decide to make it permanent, use kpatch install livepatch-test.ko, and it will add itself to startup. This allows you to reboot, and still get the patch applied automatically.
I am pretty excited, I can’t wait to get off of CentOS6, and onto the new hotness.