Network Security Model for Attack Signature Generation, Tracking and Analysis

Abstract

With the rapid growth of networks and Internet, security has become a big area of concern. The risk to the network resources is increasing day by day with the fast growing trends in attacks and intrusions. Despite current detection measures in place, timely discovery of novel attacks is still a critical issue. Intrusion Detection System (IDS) is a well known network security mechanism. IDS may use anomaly based or misuse based approach for detection of intrusions. Signature based technique is the most popular way of misuse detection. Most of IDS use signature based detection as it has low false alarm rate. But, on the contrary there is a big problem with this technique that it cannot detect unknown attacks whose signatures are not stored in their databases. Signature updating is generally a manual process and is a great overhead. Automated signature generation for Intrusion Detection Systems for proactive security of networks has been an emerging area of research. There are many solutions available in literature as proposed by various researchers. But still there is a need to address this problem as the intensity and sophistication of exploits and attacks are increasing exponentially by each passing day.

After an extensive review of literature about current security solutions and various mechanisms, a need for automatic signature generation of HTTP and SMTP attacks has been identified. The work carried out in this thesis is focused on attack detection and signature generation of unknown attacks. For the accomplishment of proposed objectives, a proactive hybrid framework, HIDESIGN (Hybrid Intrusion DEtector and SIGNature generator), for attack detection and signature generation has been designed, implemented and tested. This thesis has been organized into six chapters. These chapters include Introduction; Review of Literature; HIDESIGN: A Proposed framework for Attack Detection and Signature Generation; Implementation details of HIDESIGN; Results and Discussion; Conclusions and Future Scope.

Introduction of thesis explores several concepts and terms used in the area of security; definition and need of network security; approaches to network security, namely reactive and proactive security; different available security solutions in terms of defense mechanisms such as Firewall, Antivirus, IDS, IPS, VPN and Honeypots. In the proposed system, two mechanisms, namely, IDS and honeypots are used. A detailed description about intrusion detection, Intrusion Detection Systems and their classification based on various factors has been given. A proactive security measure called honeypot, their classification and various available honeypots are elaborated.

In Review of Literature, history and evolution of IDS along with the architectural details of two popular IDSs, namely, Snort and Bro referred in this research work has been explored. The role of anomaly detection using machine learning, available hybrid systems using misuse and anomaly based approach are discussed in this work. Thereafter, the automatic signature generation and its techniques are briefly presented. In this thesis work, in-depth review of existing signature generation mechanisms is presented by elaborating their working detail. These systems are then summarized and compared on the basis of various important parameters. Some of these systems include Honeycomb, Autograph, Earlybird, PAYL, COVERS, ARBOR, Polygraph, Nemean, Argos, Hamsa, Hancock, Nebula, Auto-Sign and F-Sign. This chapter concludes by highlighting research motivation, objectives of proposed research, major achievements and contributions of this work.

The details of HIDESIGN for automatic generation and tracking of attack signatures in context with HTTP and SMTP traffic have been discussed. Here, an introduction followed by background and attack vector details of HTTP and SMTP protocols is presented. In this research work, we have focused on server side HTTP and SMTP attacks. Various types of HTTP attacks such as SQL injection, command injection, directory traversal, cross site scripting attacks etc. have been discussed in detail. SMTP attacks and their details are also discussed in this chapter. This chapter discusses the overview, detailed architectural design and work flow of proposed system and its components. Major components of HIDESIGN include honeypot server, Misuse Detection Engine (MDE), Anomaly Detection Engine (ADE) and Signature Generation Engine (SGE). Incoming packet stream from network traffic is captured by honeypot server. Signature based detection is accomplished by MDE to detect and filter out known attacks from this traffic stream. The backbone of ADE is supervised machine learning. Any deviation from the normal behavior is detected by ADE and a malicious pool is populated. Finally, SGE generates signatures for an alert from malicious pool of data. These signatures are used to update signatures repository of IDS and may further be used to safeguard the network from intrusions.

In this thesis, the implementation details of HIDESIGN for automatic generation and tracking of attack signatures have been discussed. The deployment and experimental setup is done in a controlled environment of University’s network. Elaboration of various steps like honeypot configuration, data capturing, misuse and anomaly detection along with the procedural details with the help of PseudoCodes have been provided. Signature generation procedure and overall working of the system has also been elaborated. The proposed work has been implemented using various modules developed in Java. It makes use of honeypot technology along with Snort, a signature based IDS and anomaly detection based on supervised machine learning using Naive Bayes classifier. ADE is trained using two datasets, namely, HTTP CSIC 2010 and a subset of NSL-KDD dataset for training of HTTP and SMTP detection models, respectively.

The performance of HIDESIGN has been evaluated on the basis of results obtained as outcome of the work carried out in this thesis. The evaluation of proposed system has been performed using various evaluation metrics. These metrics have been selected in such a way that the proposed system can be tested from several aspects of good IDS. The results of various metrics like Sensitivity, Specificity, False Positive Rate, Accuracy, Detection Rate, F-Measure and ROC for both HTTP and SMTP attack detection are very encouraging. The proposed system is capable of successfully detecting and generating attack signatures of various HTTP attacks with a sensitivity and specificity of 99.97% and 98.19% for HTTP attack vectors. The sensitivity and specificity for SMTP attacks is 96.83% and 93.3%, respectively. It has reported 1.8% false positives and 0.02% false negatives for HTTP attacks whereas for SMTP attack detection 6.69% false positives and 3.17% false negatives have been reported. The auto generated signatures in Snort format have also been presented in this thesis. Further, HIDESIGN has been compared with existing recent systems proposed in literature and it has been observed that the proposed system is helpful in extending the detection capabilities of Intrusion Detection Systems by generating attack signatures in order to handle novel attacks.