中文摘要：

近年来,Shellcode攻击通常利用多态技术进行自我加密来绕过网络层设备的检测,而现有检测方法无法区分多态Shellcode与加壳保护代码.提出了一种基于双模式虚拟机的多态Shellcode检测方法,该方法改进了现有的GetPC定位机制,实现了Shellcode的初步定位,通过IA-32指令识别对网络流量的进行进一步过滤,利用有限自动机及其判别条件实现虚拟机控制流模式和数据流模式之间的切换,并通过结合现有的特征匹配技术实现对多层加密的多态Shellcode的检测.实验结果表明,针对大量真实的网络数据,该方法在保证高检测召回率的同时,能够实现对多态Shellcode与加壳保护软件的有效区分,避免了对正常流量的误报行为,并且时间开销介于静态分析与动态模拟之间,为网络层检测多态Shellcode提供了一种有效方法.

英文摘要：

Malicious code such as virus and worm propagates and attacks via the Internet which compromises network securities seriously. By exploiting vulnerabilities of services running on a remote host, the malicious code can inject shelleode into the host and gain complete control over it. Recently, shellcode attacks tend to evade network detection by employing polymorphic techniques. However, previous detection methods lack the real-time performance, resilience against evasions and the consideration of distinguishing polymorphic shellcode from normal shell protected code. We propose an enhanced polymorphic shellcode detection approach based on dual-mode virtual machine. We firstly filter network traffic via an improved GetPC location mechanism and IA-32 instruction recognition method, subsequently implement a dual-mode virtual machine to execute the target instruction flow by transferring states of the finite automaton between control-flow mode and data- flow mode according to various decision conditions. A prototype system has been implemented for our approach and experimental results indicate that when detecting real network data mixed with the polymorphic shellcode samples, the approach we proposed succeeds in differentiating the polymorphic shellcode from shell protected software without losing its accuracy and its time complexity lies between the static analysis and dynamic emulation, which makes it an encouraging method for network attack detection.