Linux Kernel System Call Fuzzing

Mgr. Arvind Rao

Master's thesis

Abstract:

Exisiting Linux kernel system call fuzzing techniques involve passing of completely random or semi-random(using templates) values as parameters to system calls to find bugs and corner cases. State of the art fuzzers successfully utilize evolutionary coverage guided techniques to improve the quality of the generated random values. Such techniques however do not detect deep bugs in the kernel. System calls, like any other API, is highly interdependent. Fuzzing such interdependent relations between the system calls require the creation of a valid context. This valid context is created when a sequence of system calls is made. The thesis builds upon the work of an existing coverage guided Linux System call fuzzer, TriforceAFL\cite{tafl}, by modifying it to support value passing in a sequence of system calls. A novel method utilizing deep learning and assisted regular expression parsing to generate high quality seeds for the modified fuzzer is also presented. Compared to the original TriforceAFL fuzzer, the modified fuzzer is able to cover more code paths and find more bugs in the kernel. The developed framework was also utilized to fuzz a proprietary DRDO Linux Kernel(DLK). The fuzzing framework was able to discover a previously undetected buffer overrun bug in the implementation.

Abstract:

Exisiting Linux kernel system call fuzzing techniques involve passing of completely random or semi-random(using templates) values as parameters to system calls to find bugs and corner cases. State of the art fuzzers successfully utilize evolutionary coverage guided techniques to improve the quality of the generated random values. Such techniques however do not detect deep bugs in the kernel. System calls, like any other API, is highly interdependent. Fuzzing such interdependent relations between the system calls require the creation of a valid context. This valid context is created when a sequence of system calls is made. The thesis builds upon the work of an existing coverage guided Linux System call fuzzer, TriforceAFL\cite{tafl}, by modifying it to support value passing in a sequence of system calls. A novel method utilizing deep learning and assisted regular expression parsing to generate high quality seeds for the modified fuzzer is also presented. Compared to the original TriforceAFL fuzzer, the modified fuzzer is able to cover more code paths and find more bugs in the kernel. The developed framework was also utilized to fuzz a proprietary DRDO Linux Kernel(DLK). The fuzzing framework was able to discover a previously undetected buffer overrun bug in the implementation.

Keywords

security, software testing, fuzzing, kernel fuzzing, coverage-guided fuzzing, system call fuzzing

Language used: English

Date on which the thesis was submitted / produced: 13. 12. 2018

Thesis defence

• Date of defence: 31. 1. 2019
• Supervisor: RNDr. Petr Švenda, Ph.D.
• Reader: Ing. Mgr. et Mgr. Zdeněk Říha, Ph.D.

Masaryk University

Faculty of Informatics

Master programme / field:
Informatics / Information Technology Security (eng.)