Thanks for linking my toot, here’s another update since then: the same person that found the race condition also found that there was a performance regression on non-synthetic benchmarks. It went super fast on his little test but is complete dogshit when used in the real world! https://lwn.net/ml/all/atI0edrmeiRuRtNQ@thinkstation/

And then he shunts the whole patchset on that person because VACATION TIME https://lwn.net/ml/all/CAHk-=wiCe2H33izZHDyDHEhcFwUr9fXZk27-WZ2vn_pn5FcZTg@mail.gmail.com/

Some kid, eh?

By the way, his argument about “you wouldn’t mention tools” is just plain wrong.

A few years back, at work, I found a security bug introduced by a commit that said something like “fix Coverity warnings”. I ended up reviewing every commit that mentioned Coverity, because that legacy codebase had a lot of rather bad tricks that triggered Coverity to have a false positive.

And when a large number of false positives were getting resolved by editing code, sometimes that resulted in a false negative. E.g. Coverity complains about a memory leak, you end up with double free. Or vice versa, it complains about double free, and you end up with a leak. (The exact example was even more on the nose; Coverity complained about use-after-free, and the fix for it introduced a real use-after-free).

Bottom line is, they do not want to have those labels because that would enable someone to actually measure how good or bad LLMs are at code.

This is another thing that is extremely familiar to most commercial developers: there’s an LLM-powered code review bot, but unlike any tool ever before, nobody’s even trying to capture statistics whether it works or not. Why? Because whoever rolls out AI tooling isn’t sure that the tooling actually works for its stated purpose, and wants to for sure get their recognition and promotion for rolling out AI tooling, without jeopardizing it with the risk of adverse KPIs.