

Using LLMs to analyze code for problems makes a lot of sense. It is just an extension of the rule based analysis we already do. It is very much a better hammer rather than a replacement of labour/thinking/taste. The programming languages we use are often difficult for humans to reason about reliably and hide a lot of errors. Our programming languages still suck (yes even Rust). If one day clang of gcc ships with a local LLM to lint code for security problems I would probably run it occasionally even on my hobby code despite being an AI hater.
If people are identifying errors using LLMs and fixing them by hand I am fine with that. It seems like a good use of the technology. If they start vibe coding drivers I want it declared so I can disable them or move to BSD.
Many projects require replicatable builds. Generative AI is driven by noise so I am not sure if it has any role in optimization or code gen. I guess we could share an RNG seed with source but honestly current tool chains are too slow as it is and ML would likely make it much worse. I want faster edit/compile/debug cycles and compilers already do insane optimisations. Ken Thompson’s Reflections on Trusting Trust becomes much scarier if your code generation is being driven by a black box of billions or parameters nobody understands. An adversary could embed anything in there.
The machine translation origins of LLMs make them really good at inspecting text and looking for things that look like known bugs. It must be a game changer for security researchers who already know what sort of problems to look for but don’t have enough hours in the day to search the haystack manually. If some of that knowledge was available in a good quality FOSS local code analysis tool it might be irresponsible not to use it. We can’t be dependent on commercial vendors and closed models.