Writing in the International Journal of Applied Cryptography, a team compares eleven leading large language models (LLMs) for software security. They found that no single system consistently outperforms its rivals in detecting vulnerabilities. This, they suggest, means organisations must select such tools according to the specific software they are analysing.
The study assessed open-source and proprietary LLMs across four public benchmark datasets covering Android applications, Internet of Things (IoT) software, and blockchain smart contracts. They also tested whether the models could identify privacy-invasive behaviour in code and whether retrieval-augmented generation (RAG), a technique that supplements an AI model with external information during use, could improve detection.
The findings come as software vulnerabilities continue to rise. Industry reports cited by the authors suggest an almost two-thirds annual increase in newly discovered vulnerabilities compared with the previous year. Vulnerabilities that have been exploited have increased by 96%. They add that software supply chain attacks, which target the software development and distribution process, have also increased sharply.
Their findings show that while several models showed promise, performance varied across datasets and domains. The authors thus argue that current LLMs remain unsuitable as universal vulnerability detectors. Limitations include outdated training data and the well-known problem of AI hallucinations, where plausible, but false, outputs are presented as fact by the AI. This, they explain, highlights the need for continued updates and testing before deployment in security-critical workflows.
Kouliaridis, V., Karopoulos, G. and Kambourakis, G. (2026) ‘Large language models for vulnerability detection: a multi-use case comparative study’, Int. J. Applied Cryptography, Vol. 5, No. 6, pp.1–17.
News media may use this press release as source material, in whole or in part, provided the content is not materially misrepresented. A link back to the original article is appreciated.
No comments:
Post a Comment