there was someone who noticed that software he used to loging was taking like a half a second longer then normal. He reported it and it turned out someone had snuck malware into the test version of a commonly used debian package...
Discussion
Ah right the xz utils
Code fragments were snuck into multiple testing branches. The commit to production brought them all together. But the xz library was chosen specifically because of its potential to spread the exploit to ssh, and 80% of the servers in the world.
But one dude who spends all his time in front of a screen noticed a subtle delay…
That’s an Easter gift as far as I’m concerned.