A Survey on Duplicate Data Filtering Methods in Big Data - P Godlin Jasil, S; Ulagamuthalvi, V (International Journal of Engineering and Technology, 2018)
Big Data analytics is the process of collecting heterogeneous huge sets of data for analyzing .The data are fetched from different sources and can be in heterogeneous form. Data arriving in the big data system will be in giga-bytes for every second. Since, the data are in huge volume, there is a possibility of redundant data that affect the network performance. This article presents the review of different filtering methods and algorithms that are used for duplicate elimination such as Bloom filter, Stable Bloom Filter, multi-layer bloom filter, Counting Bloom Filter with some disadvantages such as false positive and false negative. The aim of this paper is to propose an algorithm for eliminating the duplicate Data in a large data set by using big data analytics.Â Â