Don't use testing builds for sensitive stuff

XZ was a utility that a single person developed. Someone named Jian Tam submitted improvements to the code which Colin, the original developer implemented. Not long after, some previously unknown accounts popped up to report bugs and submit feature requests to Collin, putting pressure on him to take on a helper in maintaining the project. Jia Tan was the logical candidate

Jia Tan become more and more involved and, we now know, introduced a carefully hidden weapon into the software's source code. The revised code secretly alters another piece of software, a ubiquitous network security tool called OpenSSH, so that it passes malicious code to a target system. As a result, a specific intruder will be able to run any code they like on the target machine.

The latest version of XZ Utils, containing the backdoor, was set to be included in popular Linux distributions and rolled out across the world. However, it was caught just in time when a Microsoft engineer investigated some minor memory irregularities on his system.

Here’s the fun fact. The engineer noticed that his ssh sessions were half a second slower. He investigated and found the malicious code and was able to stop the push to all Linux distros

Somebody backdoored a widely used library. It took them years to prepare the hack and get trust of the team but it was luckily discovered by a random guy who wondered why is connection took half a second longer than usual.