Anomaly based Botnet Detection using DNS Traffic Analysis

Abstract

Cybercrimes are evolving on a regular basis and these crimes are becoming a greater threat day by day. Earlier these threats were very general and unorganized. In the last decade, these attacks have become highly sophisticated in nature. This higher level of coordination is possible mainly due to Botnet which is a cluster of infected hosts controlled remotely by an attacker (Botmaster). The number of infected machines is continuously rising thereby resulting in Botnets with many of these having even over a million infected machines. This innumerous set of machines with varied computational and storage capabilities give the botmaster a lethal weapon to launch various security attacks. This never-ending menace of the botnet is causing many serious problems on the Internet. Domain Name System is a large-scale distributed database on the Internet, which is being abused as a Botnet communication channel. Significant efforts have been made in detecting botnet at the global level which relies heavily on finding failed queries and domain flux information for botnet detection, there are very few efforts being made to detect bot infection at an enterprise level. Detecting bot-infected machines are vital for any organization in combating various security threats. This research work proposes a novel anomaly-based detection technique which considers captured DNS traffic from LAN hosts on hourly basis to generate DNS fingerprint and attempts to find anomalous behavior which is quite different from normal machine behavior. This research work successfully demonstrates the DNS Anomaly Detection (named BotDAD) technique for detecting bot-infected machine in a network using DNS fingerprinting. It uses a feature extractor module to extract DNS attributes and build a host profile for all hosts in the network. The host profile is then parsed to generate DNS fingerprint. BotDAD creates DNS fingerprint of each host in the network and uses anomaly detection engine to label them as bot or clean. BotDAD uses a machine learning classifier to develop a trained model for future predictions. The system is evaluated against DNS network traffic captured from TIET Patiala campus on an hourly basis. The system was able to detect Bot infected machines in the network. The domains used for C&C by these bots were validated against online DGA domains database. Results from BotDAD gives an accuracy of 0.9978. To improve the accuracy of the BotDAD, a multi-layer neural network named DeepDAD was implemented. DeepDAD is a Deep Learning based DNS Anomaly Detection tool created as part of this research which considers multipoint anomaly detection and uses deep learning algorithms. Instead of relying on a single point anomaly for labeling DNS fingerprint as malicious, an improved labeling technique which uses multipoint anomaly detection is implemented. Two machine learning frameworks namely Scikit-learn and TensorFlow were used to train and test the model showing significant improvement over the results obtained using BotDAD. Finally, a graphical user interface for easy testing and comparison is presented.