Optimizing Storage Using Clustering Technique for Tick data

Abstract

The clustering problem widespread in day-to-day life where data can be found, mined, or generated for most situations imaginable. To manage huge amount of data, it is require to group the similar type data. As the number of possible data sets grows and the data sets become larger in both number of data points and variables, the automation of this process through clustering algorithms is increasingly important. Tick data is data generated by various applications periodically that is why it is require keeping track the values changing over time and also requiring optimizing redundant data to reduce storage space. Here in this thesis, our aim is to optimize the storage space using clustering technique and to compute time complexity of propose method. The approach starts with k partitions of tick dataset. The partitions are based on the columns of tick data. After the partition the number of clusters is obtained and then merges the clusters and finally the clusters are obtained in the normalized form. The next step is to construct binary indicator vector that contains binary information generated after matching two concurrent columns and rows. The algorithm also counts the zeroes and ones that occur in the tick data. The next step is to eliminate all the rows which are having duplicate values. The propose approach also compute the compression ratio and execution time that varies as per the number of clusters selected and system configuration. Performance analysis in terms of execution time in seconds varies as per the number of clusters selected and system configuration.