STAF: Leveraging LLMs for Automated Attack Tree-Based Security Test Generation
Abstract
This thesis addresses the critical challenge of automating security test case generation from attack trees, a process traditionally labor-intensive and lacking comprehensive automation in software testing. We introduce the Security Test Automation Framework (STAF), a novel approach leveraging Large Language Models (LLMs) and a two-step self-corrective Retrieval-Augmented Generation (RAG) framework. STAF provides an end-to-end solution for automatically generating executable security test cases from attack trees, enabling comprehensive coverage of potential vulnerabilities and attack vectors within software systems. Our methodology employs a custom RAG framework designed specifically for security test case generation, addressing limitations in existing approaches. Experimental results demonstrate that STAF augmented LLama 3.1 \& Qwen2.5 outperform closed-source models like GPT-4o and Claude 3.5 Sonnet, despite having 2-3 times fewer parameters. Additionally, we present the first publicly available benchmark dataset for security test case generation from attack trees, supporting standardized evaluation. The study reveals significant improvements in efficiency, accuracy, and scalability, with STAF seamlessly integrating into existing workflows. These findings mark a substantial advancement in security testing methodologies, potentially transforming how organizations approach vulnerability assessment and mitigation in software systems.