Pairing Security Advisories with Vulnerable Functions Using Open-Source LLMs

Trevor Dunlap, John Speed Meyers, Bradley Reaves, and William Enck

Conference on Detection of Intrusions and Malware and Vulnerability Assessment, 2024

Open-source LLMs can pinpoint which functions a security patch fixes, achieving 173% higher precision than treating all changed functions as vulnerable.

Abstract

As the reliance on open-source software dependencies increases, managing the security vulnerabilities in these dependencies becomes complex. State-of-the-art industry tools use reachability analysis of code to alert developers when security vulnerabilities in dependencies are likely to impact their projects. These tools heavily rely on precisely identifying the location of the vulnerability within the dependency, speciﬁcally vulnerable functions. However, the process of identifying vulnerable functions is currently either manual or uses a naive automated approach that falsely assumes all changed functions in a security patch link are vulnerable. In this paper, we explore using open-source large language models (LLMs) to improve pairing security advisories with vulnerable functions. We explore various prompting strategies, learning paradigms (i.e., zero-shot vs. few-shot), and show our approach generalizes to other open-source LLMs. Compared to the naive automated approach, we show a 173% increase in precision while only having an 18% decrease in recall. The signiﬁcant increase in precision to enhance vulnerable function identiﬁcation lays the groundwork for downstream techniques that depend on this critical information for security analysis and threat mitigation.

Citation (IEEE)

T. Dunlap, J. S. Meyers, B. Reaves, and W. Enck, “Pairing Security Advisories with Vulnerable Functions Using Open-Source LLMs,” in Conference on Detection of Intrusions and Malware and Vulnerability Assessment, 2024.

BibTeX

@inproceedings{dmre24,
  author = {Dunlap, Trevor and Meyers, John Speed and Reaves, Bradley and Enck, William},
  booktitle = {Conference on Detection of Intrusions and Malware and Vulnerability Assessment},
  date = {2024-07},
  title = {Pairing Security Advisories with Vulnerable Functions Using Open-Source {LLMs}},
}