The XZ Utils backdoor in Linux points to a wider supply chain security issue, and we need more than community spirit to keep it at bay

Published Apr 11, 2024
by Pieter Danhieux
cASE sTUDY

The XZ Utils backdoor in Linux points to a wider supply chain security issue, and we need more than community spirit to keep it at bay

Published Apr 11, 2024
by Pieter Danhieux
View Resource
View Resource

The cybersecurity industry was once again placed on high alert following the discovery of an insidious software supply chain compromise. The vulnerability, affecting the XZ Utils data compression library that ships with major Linux distributions, is logged under CVE-2024-3094 and boils down to a backdoor deliberately inserted by a once-trusted volunteer system maintainer. Allowing remote code execution (RCE) in some instances if successfully exploited, it represents a high-severity issue with the ability to cause serious damage in established software build processes.

Thankfully, another maintainer discovered this threat before the malicious code entered stable Linux releases, but it still poses an issue to those who have started running version 5.6.0. and 5.6.1. of XZ Utils as part of Fedora Rawhide, and organizations are urged to patch it as an emergency-level priority. If this discovery were not made in time, the risk profile would make it one of the most devastating supply chain attacks on record, perhaps even eclipsing SolarWinds.

The reliance on community volunteers to maintain critical systems is widely documented, yet rarely discussed until high-impact issues such as this incident bubble to the surface. While their tireless work is essential to the maintenance of open-source software, this highlights the need for serious emphasis on security skills and awareness at the developer level, not to mention heightened access controls around software repos.

What is the XZ Utils backdoor, and how is it mitigated?

On March 29, Red Hat published an urgent security alert to inform users of Fedora Linux 40 and Fedora Rawhide that the latest versions of the “XZ” compression tools and libraries contain malicious code that appears to be purpose-built to facilitate unauthorized third-party access.How this malicious code was injected will likely be the subject of intense study in the future, but it amounts to a sophisticated, patient, years-long social engineering exercise on the part of the threat actor, a pseudonymous attacker called “Jia Tan”. This individual spent countless hours gaining the trust of other maintainers, making legitimate contributions to the XZ Utils project and community for over two years, eventually earning “trusted maintainer” status after multiple sockpuppet accounts eroded confidence in volunteer project owner, Lasse Collin:

Jia Tan inserting himself as a contributor to the project. Source: Mail Archive.

The original maintainer being overworked. Jia Tan gaining more community confidence to take over. Source: Mail Archive.

This unusual scenario is a prime example of a highly technical person still falling victim to tactics typically reserved for use against those who are less savvy, showing the need for precision, role-based security awareness training. It was only for the inquisitiveness and quick thinking of Microsoft software engineer and PostgreSQL maintainer, Andres Freund, that the backdoor was discovered and versions rolled back, thus stopping what could have been the most devastating supply chain attack in recent memory.

The backdoor itself is officially being tracked as a vulnerability of the highest possible severity in NIST’s registry. Initially thought to allow SSH authentication bypassing, further investigation revealed that it enabled unauthenticated remote code execution on vulnerable Linux systems, including Fedora Rawhide, Fedora 41, Kali Linux, openSUSE MicroOS, openSUSE Tumbleweed, and some versions of Debian.

Jia Tan appears to have gone to great lengths to obfuscate the malicious package, which, when triggered to construct itself during the build process, hampers authentication in SSHd via systemd. As Red Hat detailed, in the right circumstances, this interference could potentially allow an attacker to break SSHd authentication and gain remote unauthorized access to the entire system.

Jia Tan’s first commit in the libarchive repo. Replacing the safe_fprintf() with function fprintf(). The intention may not have been malicious at this stage, but it can't be overlooked that this change can potentially introduce a character escape vulnerability. Source: GitHub.



Microsoft, among others, has published comprehensive guidance on scanning systems for instances of the exploit and mitigating its effect, and the immediate action recommended by CISA is that affected developers and users should downgrade XZ Utils to an uncompromised version, like XZ Utils 5.4.6 Stable.

Preventing this attack type is incredibly difficult - especially when utilizing open-source components in software - as there is precious little assurance and transparency over the security of the supply chain. We've already combatted accidental flaws in the software supply chain, but this risk has elevated to include security bugs deliberately planted with malice to compromise open-source security.

Most developers will not be able to stop an attack of this nature unless they have a strong sense of security awareness, healthy security knowledge, and a sprinkling of paranoia. It’s almost a case of requiring a threat actor mindset. However, a chief consideration should always center around source code repos that are controlled internally (i.e. not open-source). These should only be accessible to people who have verified, relevant security skills. AppSec professionals might consider a setup like branch-level security controls, only allowing security-skilled developers to commit changes to the final master branch.

Volunteer maintainers are heroes, but it (should) take a village to sustain secure software

To those outside the realm of software engineering, the notion that a spirited community of volunteers painstakingly maintains critical systems in their own time is a difficult concept to grasp, but this is the nature of open-source development, and it continues to be an area of critical risk for security professionals protecting the supply chain.

Open-source software is a vital part of virtually every enterprise’s digital ecosystem, and trusted maintainers (of which most are acting in good faith) are truly heroic in their selfless pursuit of technological progress and integrity, but it is farcical to keep them delivering in isolation. In these DevSecOps-centric times, security is a shared responsibility, and every developer must be armed with the knowledge and right-fit tooling to navigate the security issues they are likely to encounter in their workday. Security awareness and hands-on skills should be non-negotiable in the software development process, and it’s up to security leaders to influence change at the enterprise level. 

Build a thriving security culture in your organization today with in-depth Courses from Secure Code Warrior.

View Resource
View Resource

Author

Pieter Danhieux

Pieter Danhieux is a globally recognized security expert, with over 12 years experience as a security consultant and 8 years as a Principal Instructor for SANS teaching offensive techniques on how to target and assess organizations, systems and individuals for security weaknesses. In 2016, he was recognized as one of the Coolest Tech people in Australia (Business Insider), awarded Cyber Security Professional of the Year (AISA - Australian Information Security Association) and holds GSE, CISSP, GCIH, GCFA, GSEC, GPEN, GWAPT, GCIA certifications.

Want more?

Dive into onto our latest secure coding insights on the blog.

Our extensive resource library aims to empower the human approach to secure coding upskilling.

View Blog
Want more?

Get the latest research on developer-driven security

Our extensive resource library is full of helpful resources from whitepapers to webinars to get you started with developer-driven secure coding. Explore it now.

Resource Hub

The XZ Utils backdoor in Linux points to a wider supply chain security issue, and we need more than community spirit to keep it at bay

Published Apr 11, 2024
By Pieter Danhieux

The cybersecurity industry was once again placed on high alert following the discovery of an insidious software supply chain compromise. The vulnerability, affecting the XZ Utils data compression library that ships with major Linux distributions, is logged under CVE-2024-3094 and boils down to a backdoor deliberately inserted by a once-trusted volunteer system maintainer. Allowing remote code execution (RCE) in some instances if successfully exploited, it represents a high-severity issue with the ability to cause serious damage in established software build processes.

Thankfully, another maintainer discovered this threat before the malicious code entered stable Linux releases, but it still poses an issue to those who have started running version 5.6.0. and 5.6.1. of XZ Utils as part of Fedora Rawhide, and organizations are urged to patch it as an emergency-level priority. If this discovery were not made in time, the risk profile would make it one of the most devastating supply chain attacks on record, perhaps even eclipsing SolarWinds.

The reliance on community volunteers to maintain critical systems is widely documented, yet rarely discussed until high-impact issues such as this incident bubble to the surface. While their tireless work is essential to the maintenance of open-source software, this highlights the need for serious emphasis on security skills and awareness at the developer level, not to mention heightened access controls around software repos.

What is the XZ Utils backdoor, and how is it mitigated?

On March 29, Red Hat published an urgent security alert to inform users of Fedora Linux 40 and Fedora Rawhide that the latest versions of the “XZ” compression tools and libraries contain malicious code that appears to be purpose-built to facilitate unauthorized third-party access.How this malicious code was injected will likely be the subject of intense study in the future, but it amounts to a sophisticated, patient, years-long social engineering exercise on the part of the threat actor, a pseudonymous attacker called “Jia Tan”. This individual spent countless hours gaining the trust of other maintainers, making legitimate contributions to the XZ Utils project and community for over two years, eventually earning “trusted maintainer” status after multiple sockpuppet accounts eroded confidence in volunteer project owner, Lasse Collin:

Jia Tan inserting himself as a contributor to the project. Source: Mail Archive.

The original maintainer being overworked. Jia Tan gaining more community confidence to take over. Source: Mail Archive.

This unusual scenario is a prime example of a highly technical person still falling victim to tactics typically reserved for use against those who are less savvy, showing the need for precision, role-based security awareness training. It was only for the inquisitiveness and quick thinking of Microsoft software engineer and PostgreSQL maintainer, Andres Freund, that the backdoor was discovered and versions rolled back, thus stopping what could have been the most devastating supply chain attack in recent memory.

The backdoor itself is officially being tracked as a vulnerability of the highest possible severity in NIST’s registry. Initially thought to allow SSH authentication bypassing, further investigation revealed that it enabled unauthenticated remote code execution on vulnerable Linux systems, including Fedora Rawhide, Fedora 41, Kali Linux, openSUSE MicroOS, openSUSE Tumbleweed, and some versions of Debian.

Jia Tan appears to have gone to great lengths to obfuscate the malicious package, which, when triggered to construct itself during the build process, hampers authentication in SSHd via systemd. As Red Hat detailed, in the right circumstances, this interference could potentially allow an attacker to break SSHd authentication and gain remote unauthorized access to the entire system.

Jia Tan’s first commit in the libarchive repo. Replacing the safe_fprintf() with function fprintf(). The intention may not have been malicious at this stage, but it can't be overlooked that this change can potentially introduce a character escape vulnerability. Source: GitHub.



Microsoft, among others, has published comprehensive guidance on scanning systems for instances of the exploit and mitigating its effect, and the immediate action recommended by CISA is that affected developers and users should downgrade XZ Utils to an uncompromised version, like XZ Utils 5.4.6 Stable.

Preventing this attack type is incredibly difficult - especially when utilizing open-source components in software - as there is precious little assurance and transparency over the security of the supply chain. We've already combatted accidental flaws in the software supply chain, but this risk has elevated to include security bugs deliberately planted with malice to compromise open-source security.

Most developers will not be able to stop an attack of this nature unless they have a strong sense of security awareness, healthy security knowledge, and a sprinkling of paranoia. It’s almost a case of requiring a threat actor mindset. However, a chief consideration should always center around source code repos that are controlled internally (i.e. not open-source). These should only be accessible to people who have verified, relevant security skills. AppSec professionals might consider a setup like branch-level security controls, only allowing security-skilled developers to commit changes to the final master branch.

Volunteer maintainers are heroes, but it (should) take a village to sustain secure software

To those outside the realm of software engineering, the notion that a spirited community of volunteers painstakingly maintains critical systems in their own time is a difficult concept to grasp, but this is the nature of open-source development, and it continues to be an area of critical risk for security professionals protecting the supply chain.

Open-source software is a vital part of virtually every enterprise’s digital ecosystem, and trusted maintainers (of which most are acting in good faith) are truly heroic in their selfless pursuit of technological progress and integrity, but it is farcical to keep them delivering in isolation. In these DevSecOps-centric times, security is a shared responsibility, and every developer must be armed with the knowledge and right-fit tooling to navigate the security issues they are likely to encounter in their workday. Security awareness and hands-on skills should be non-negotiable in the software development process, and it’s up to security leaders to influence change at the enterprise level. 

Build a thriving security culture in your organization today with in-depth Courses from Secure Code Warrior.

We would like your permission to send you information on our products and/or related secure coding topics. We’ll always treat your personal details with the utmost care and will never sell them to other companies for marketing purposes.

Submit
To submit the form, please enable 'Analytics' cookies. Feel free to disable them again once you're done.