中文摘要：

由于Hadoop自身并不适合海量小文件处理，目前的重复数据删除方法主要基于文件的二进制特征，无法识别经过信号处理后的同一首歌曲，也不能满足海量数据在线处理的要求。提出一种采用声学指纹去重的海量MP3文件存储架构，结合音乐文件自身的声学特性和MP3文件包含的元信息，通过索引、在线归并和NAF去重，很好地解决了小文件过多时内存瓶颈问题，同时提供了更好的去重效果；离线归并和副本调整模块根据系统的运行状况不断优化存储。实验结果表明，该架构在性能、去重率、可管理性和可扩展性方面达到了良好的平衡，极大地提高了去重率，与可变分块CDC相比，去重率提高了100％，具有良好的实用价值。

英文摘要：

Due to the Hadoop itself is not suitable for processing of the mass of small files. And current data de-duplication methods are mainly based on the binary characteristics of the file, so it cannot recognize the same song after the signal process- ing and also cannot meet the requirements of the online processing of massive data. This paper presented a de-duplication stor- age architecture of the mass of the MP3 file based on the acoustic fingerprint. It combined with music files on the acoustic char- acteristics and the recta-information of MP3 files, de-duplication by index, merge online and NAF, solved the memory bottle- neck problem effectively in the face of too many small files. At the same time it provided a better de-duplication effect. Offline merge and the replication place module optimized storage continually according to the operating conditions of the system. The experimental results show that the architecture can achieve a good balance on performance, the rate of de-duplication, manage- ability and scalability.