Developing a Cybersecurity Domain Chatbot based on an Open Source Large Language Model

Shahrukh Azhar AHSAN

Master's thesis

Abstract:

The objective of this research is to determine the effectiveness of fine-tuned open-source LLMs in the domain of cybersecurity. Specifically, the study evaluates how effective fine-tuning is for LLMs to learn and provide accurate information about recently reported software vulnerabilities. The LLMs used in this study were Falcon-7B and Llama-2-7b-chat-hf. A custom dataset of 19,135 question-answer pairs referencing publicly reported software vulnerabilities in 2023, was used. These vulnerabilities were sourced from the NVD (National Vulnerability Database) and OWASP (The Open Web Application Security Project). A total of six fine-tuned models were obtained. The similarity between their generated answers and true answers was determined with gpt-3.5 for evaluation. The best fine-tuned models showed quite significant improvements in validation accuracy and code generation capabilities w.r.t their base variants.

Abstract:

The objective of this research is to determine the effectiveness of fine-tuned open-source LLMs in the domain of cybersecurity. Specifically, the study evaluates how effective fine-tuning is for LLMs to learn and provide accurate information about recently reported software vulnerabilities. The LLMs used in this study were Falcon-7B and Llama-2-7b-chat-hf. A custom dataset of 19,135 question-answer pairs referencing publicly reported software vulnerabilities in 2023, was used. These vulnerabilities were sourced from the NVD (National Vulnerability Database) and OWASP (The Open Web Application Security Project). A total of six fine-tuned models were obtained. The similarity between their generated answers and true answers was determined with gpt-3.5 for evaluation. The best fine-tuned models showed quite significant improvements in validation accuracy and code generation capabilities w.r.t their base variants.

Keywords

machine learning, large language models, cybersecurity, open-source LLMs, LoRA, QLoRA, natural language processing, transformers, fine-tuning

Language used: English

Date on which the thesis was submitted / produced: 20. 8. 2024

Thesis defence

• Supervisor: prof. Dr. Michael Heigl

UNIVERSITY OF SOUTH BOHEMIA IN ČESKÉ BUDĚJOVICE

Faculty of Science

Master programme / field:
Artificial Intelligence and Data Science / Artificial Intelligence and Data Science