Below is the original text I wrote for a white paper on how CloudLinux’s KernelCare product live-patches software libraries.
Patching Shared Libraries with KernelCare+
Shared libraries present serious vulnerabilities.
To operate web servers securely, it’s not enough to patch their kernels—their software libraries must be patched as well. Otherwise, they’re vulnerable to attacks that exploit vulnerabilities such as:
- Heartbleed. Attackers exploiting this OpenSSL request validation flaw could read a server’s memory, then gain control of it. Immediately after it became known, it was used to steal the hospital records of 4.5 million patients. This vulnerability still exists on many systems, even though patches for it have been available since 2014.
- GHOST. Attackers exploiting this glibc buffer overflow flaw could use gethostby* functions to make network requests that enabled them to gain control of a server. MySQL servers, Exim, and other mail servers were vulnerable to it. Once it became known, enterprises worldwide scrambled to patch it before it could be exploited.
OpenSSL and glibc continue to present security problems on Linux systems. As of 2020:
- Attacks on OpenSSL accounted for 71% of vulnerabilities targeted in the technology industry.
- glibc still handled memory operations in a way that attackers could use to execute malicious code.
Vulnerabilities like this are why almost one in five attacks target OpenSSL, but it’s not just OpenSSL and glibc that put Linux servers at risk. Libarchive, a compression library included by default in a vast number of Linux distributions and software utilities, contains a vulnerability that can enable attackers to execute code on remote servers.
These sorts of library vulnerabilities are being discovered at an increasing rate: from 2017 to 2019, they nearly doubled in number. They’re also becoming more widespread: In 2020, critical vulnerabilities known as Ripple20 were discovered in a widely-used TCP/IP library, exposing hundreds of millions of internet-connected devices to attack.
Patching libraries through reboots is problematic.
Enterprises usually address library vulnerabilities by rebooting their servers. Sysadmins often don’t know which services use which libraries, so they just reboot the whole server to update them all. These reboots, however, bring serious problems:
- Server downtime. When servers are down, web sites go down, and display only error messages to visitors. After rebooting, it can take some time for server performance to stabilize, and occasionally servers don’t come back up properly after a reboot.
- Windows of vulnerability. Because rebooting is laborious and problematic, enterprises often only do it on a periodically scheduled basis, leaving their servers open to attack. Even if they reboot every 30 days to comply with security standards, their servers may be vulnerable for two weeks or more.
Even if they’re patched manually, shared libraries may contain vulnerabilities. When libraries are updated on disk, old unpatched files can persist in a server’s memory. What’s more, vulnerability scanners don’t detect these files.
KernelCare+ patches libraries without rebooting.
Just like KernelCare, KernelCare+ patches the Linux kernel. It differs from KernelCare in that it patches shared libraries as well.
KernelCare+ patches shared libraries and detects library-related vulnerabilities. It even patches library files in memory, and does this in a way that makes reboots unnecessary.
Right now, KernelCare+ patches the glibc and OpenSSL libraries, because these are the ones most often attacked. In the future, it will patch more shared libraries, such as those related to PHP and Python.
It employs sophisticated patching technology.
To patch shared libraries on web servers, KernelCare+ employs an innovative four-stage patching process:
- The patch is created by the KernelCare team.
A library’s source code–both original and patched–are translated into assembly language. These files are compared, and the new patched code is put in a new section of the same ELF file. After the code is compiled and linked, the patch is extracted from the resulting binaries. The patch files are then extracted from the ELF sections. - The patch is uploaded to the patch server.
The binary files are treated as a single patch, which is then uploaded to a dedicated KernelCare+ patch server. The patch server then distributes the patch to customers’ servers. - The patch is downloaded to the local agent.
An agent program on each local server, lcarectl, “talks to” the patch server, which looks for known libraries on the local server. The agent program then downloads the patch needed for each library present on the local server. - The patch is applied to the local server.
Using Linux APIs, memory near a library is allocated, and the patch is copied into it. Then the agent program reroutes calls from old code to the new patched versions via unconditional jumps. Once this patching process is complete, the local server’s libraries are fully protected against all known attacks.
See for yourself how it keeps servers safe.
To learn more about KernelCare+, and evaluate it in your environment with a free 30-day trial, visit www.kernelcare.com.