Selected botnet detection techniques using flow data

Mgr. Tanay Bhattacharya

Master's thesis

Abstract:

The botnet is one of the most widespread malware in the wild aimed at coordinated compromise, control, and misutilization of vulnerable machines. These compromised machines are known as zombies/bots and the controller is often called botmaster. These machines are further used as launching platform to carry out coordinated attack on target system(s)/network. In general, depending on the characteristics of bots, different techniques have been proposed to carryout their detection and bring down such coordinated attack by Security Researchers/Vendors. To counter this, bot writers have come up with detection evasion techniques such as Encrypted Communication, Fast Flux, Domain Generation Algorithm (DGA), migration from earlier IRC based communication to recent Peer to Peer (P2P) techniques etc. It is a catch and run game between bot writers and bot detectors and so far there seems to be no clear winner! My research will concentrate mostly on Analysis and Implementation of selected bot detection methods that are independent of common detection evasive techniques. I will be using Flow based bot detection methodologies that generally consider OSI Layer 3 and 4 metadata without payload profiling for IRC/HTTP/P2P/DNS based botnets. Different existing botnet detection solutions will be considered and some of them will be implemented in experimental setup and tested with live bots and various other Datasets. The performance of such solutions will be evaluated. This will help to figure out optimum botnet detection parameters and techniques for different botnet types.

Abstract:

The botnet is one of the most widespread malware in the wild aimed at coordinated compromise, control, and misutilization of vulnerable machines. These compromised machines are known as zombies/bots and the controller is often called botmaster. These machines are further used as launching platform to carry out coordinated attack on target system(s)/network. In general, depending on the characteristics of bots, different techniques have been proposed to carryout their detection and bring down such coordinated attack by Security Researchers/Vendors. To counter this, bot writers have come up with detection evasion techniques such as Encrypted Communication, Fast Flux, Domain Generation Algorithm (DGA), migration from earlier IRC based communication to recent Peer to Peer (P2P) techniques etc. It is a catch and run game between bot writers and bot detectors and so far there seems to be no clear winner! My research will concentrate mostly on Analysis and Implementation of selected bot detection methods that are independent of common detection evasive techniques. I will be using Flow based bot detection methodologies that generally consider OSI Layer 3 and 4 metadata without payload profiling for IRC/HTTP/P2P/DNS based botnets. Different existing botnet detection solutions will be considered and some of them will be implemented in experimental setup and tested with live bots and various other Datasets. The performance of such solutions will be evaluated. This will help to figure out optimum botnet detection parameters and techniques for different botnet types.

Keywords

Botnet, Netflow, Machine Learning, Classification, IRC/HTTP/P2P, Command and Control(C2), Anomaly based, Domain Generation Algorithm(DGA), Fast Flux.

Language used: English

Date on which the thesis was submitted / produced: 4. 1. 2017

Thesis defence

• Date of defence: 10. 2. 2017
• Supervisor: Ing. Mgr. et Mgr. Zdeněk Říha, Ph.D.
• Reader: RNDr. Radim Ošťádal

Masaryk University

Faculty of Informatics

Master programme / field:
Informatics / Information Technology Security (eng.)