针对配电自动化监控系统中大数据集信息和压缩存储率低的问题,提出一种基于Hadoop云计算的信息流集群无损压缩新方法;从监控业务信息流出发,将Lzo无损压缩编码融人Map/Reduce计算任务中,实现对大数据集监控信息流的有效压缩;以配电网监控信息流的处理为例,分别取300万和1000万断面记录进行测试,结果表明:每秒可压缩处理记录数从4万提高至8.4万条,处理效率提高了52.4%,同时压缩比也从55.4%提高至62%,适用于动态量测过程的信息处理,提高了压缩处理的效果。
In order to solve low efficiency problem of large data sets compression storage in distribution network dispatching monitoring system, a new processing method based on cloud computing Hadoop is proposed for information flow lossless compression. The key business information flow is extracted from monitoring system. Then the Lzo lossless compression codes fused into Map/Reduce parallel computation task, which achieving effective compression for large data sets. Taking distribution network monitoring information flows as test example. Result shows that when section measurement record sets increased from 300 million to 1000 million, the compressed record per second is in creased from 4 million to 8.4 million, and the compression ratio from 55.4% to 62%. The processing efficiency is improved by 52.4%. New compression method is faster and more suitable for processing dynamic measurement and control process information, which improves the compression effect.