Flow-based Analysis of DTLS Protocol

Mgr. Akhilesh Kumar Soni

Master's thesis

Abstract:

This thesis work presents the operation and analysis of Datagram Transport Layer Security (DTLS) protocol between the client and the server during the course of provisioning of encrypted traffic. It helps in DTLS monitoring in high-speed and large-scale network environments in the continuously increasing Internet traffic. It is, therefore, apparent to deploy advanced monitoring techniques to provide long-term network visibility in near real-time. In this work, a flow-based measurement system architecture to measure DTLS protocol is proposed. The measurement system architecture comprises of the Flowmon devices along with passive network monitoring support infrastructure. The collected flow data can further be utilised for the behavioural analysis of network to indicate legitimate and malicious traffic, analyse and plan or improve the effective network defence mechanism, proving cyber threats etc. Integration of flow-based monitoring to an early warning system could ensure situational awareness of the network. The flow-based network monitoring presented in this thesis is passive type of network monitoring, which allows detecting the threats without being noticed by attackers. Therefore, it improves the effectiveness and speed of response to the network threats when integrated to the network threat response mechanisms. The functioning of the Flowmon exporter and its application environment for creating plug-ins is also described. The work also comprises the design and implementation of a tool-set for gathering and processing useful information from an encrypted communication in the form of a plugin module for the Flowmon exporter/probe. The last part deals with the measurement verification along with the proposal of DTLS use-cases.

Abstract:

This thesis work presents the operation and analysis of Datagram Transport Layer Security (DTLS) protocol between the client and the server during the course of provisioning of encrypted traffic. It helps in DTLS monitoring in high-speed and large-scale network environments in the continuously increasing Internet traffic. It is, therefore, apparent to deploy advanced monitoring techniques to provide long-term network visibility in near real-time. In this work, a flow-based measurement system architecture to measure DTLS protocol is proposed. The measurement system architecture comprises of the Flowmon devices along with passive network monitoring support infrastructure. The collected flow data can further be utilised for the behavioural analysis of network to indicate legitimate and malicious traffic, analyse and plan or improve the effective network defence mechanism, proving cyber threats etc. Integration of flow-based monitoring to an early warning system could ensure situational awareness of the network. The flow-based network monitoring presented in this thesis is passive type of network monitoring, which allows detecting the threats without being noticed by attackers. Therefore, it improves the effectiveness and speed of response to the network threats when integrated to the network threat response mechanisms. The functioning of the Flowmon exporter and its application environment for creating plug-ins is also described. The work also comprises the design and implementation of a tool-set for gathering and processing useful information from an encrypted communication in the form of a plugin module for the Flowmon exporter/probe. The last part deals with the measurement verification along with the proposal of DTLS use-cases.

Keywords

DTLS, Datagram TLS, Encrypted Traffic Analysis (ETA), DTLS Flow Analysis (DFA), Secure UDP (UDP-S), Flowmon Export

Language used: English

Date on which the thesis was submitted / produced: 13. 12. 2018

Thesis defence

• Date of defence: 1. 2. 2019
• Supervisor: doc. Ing. Pavel Čeleda, Ph.D.
• Reader: RNDr. Petr Velan, Ph.D.

Masaryk University

Faculty of Informatics

Master programme / field:
Informatics / Information Technology Security (eng.)