Analysis, Anti-Analysis, Anti-Anti-Analysis: An Overview of the Evasive Malware Scenario

  • Marcus Botacin UNICAMP
  • Vitor Falcão da Rocha UNICAMP
  • Paulo Lício de Geus UNICAMP
  • André Grégio UFPR


Malicious programs are persistent threats to computer systems, and their damages extend from financial losses to critical infrastructure attacks. Malware analysis aims to provide useful information to be used for forensic procedures and countermeasures development. To thwart that, attackers make use of anti-analysis techniques that prevent or difficult their malware from being analyzed. These techniques rely on instruction side-effects and that system's structure checks are inspection-aware. Thus, detecting evasion attempts is an important step of any successful investigative procedure. In this paper, we present a broad overview of what anti-analysis techniques are being used in malware and how they work, as well as their detection counterparts, i.e., the anti-anti-analysis techniques that may be used by forensic investigators to defeat evasive malware. We also evaluated over one hundred thousand samples in the search of the presence of anti-analysis technique and summarized the obtained information to present an evasion-aware malware threat scenario.


Bachaalany, E. (2005). Detect if your program is running inside a virtual machine. [link].

Barngert, J. (2013). Trapcc.

Botacin, M., Grégio, A., and de Geus, P. (2015). Uma visão geral do malware ativo no espaço nacional da internet entre 2012 e 2015. Anais do XV SBSEG.

Branco, R. R., Barbosa, G. N., and Neto, P. D. (2012). Scientific but not academical overview of malware anti-debugging, anti-disassembly and anti- vm technologies.

Bremer, J. (2012). Ssexy.

Calvet, J., Fernandez, J. M., and Marion, J.-Y. (2012). Aligot: Cryptographic function identification in obfuscated binary programs. In Proc. of the 2012 ACM Conf. on Computer and Communications Security, CCS ’12, pages 169–182, New York, NY, USA. ACM. (2015). Estatísticas dos incidentes reportados ao Access Date: May/2017.

Dinaburg, A., Royal, P., Sharif, M., and Lee,W. (2008). Ether: Malware analysis via hardware virtualization extensions. In Proc. of the 15th ACM Conf. on Computer and Communications Security, CCS ’08, pages 51–62, New York, NY, USA. ACM.

domas (2015). Movfuscator.

Eliam, E. (2005). Reverse: Secrets of Reverse Engineering. Willey.

Ferrand, O. (2015). How to detect the cuckoo sandbox and to strengthen it? Journal of Computer Virology and Hacking Techniques, 11(1):51–58.

Ferrie, P. (2007). Attacks on virtual machine emulators.

Ferrie, P. (2008). Anti-unpacker tricks.

hexacorn (2014). Protecting vmware from cpuid hypervisor detection. [link].

Hexacorn (2014). Rdtscp - a recooked antire trick.

Kruegel, C., Kirda, E., and Moser, A. (2007). Limits of Static Analysis for Malware Detection. In Proc. of the 23rd Annual Computer Security Applications Conference (ACSAC) 2007.

Laboratory, B. (2012). Detecting vmware.

Lee, J., Kang, B., and Im, E. G. (2013). Rule-based anti-anti-debugging system. In Proc. of the 2013 Research in Adaptive and Convergent Systems, RACS ’13, pages 353–354, New York, NY, USA. ACM.

Lindorfer, M., Kolbitsch, C., and Milani Comparetti, P. (2011). Detecting environment-sensitive malware. In Proc. of the 14th International Conf. on Recent Advances in Intrusion Detection, RAID’11, pages 338–357, Berlin, Heidelberg. Springer-Verlag.

Liu, Q. (2012). Just another malware analyzer.

Lyashko, A. (2011). Stealth import of windows api.

Microsoft (2016). Isdebuggerpresent. [link].

Microsoft (2017a). Download detours express.

Microsoft (2017b). Getprocessheap function. [link].

Microsoft (2017c). Peb structure. [link].

Microsoft (2017d). Peb_ldr_data structure. [link].

Microsoft (2017e). Setwindowshookex function. [link].

Microsoft (2017f). Virtualquery function. [link]. (2006). The torpig/anserin/sinowal family of trojans detect virtual machine information and abort infection when present.

Nasi, E. (2014). Bypass antivirus dynamic analysis.

Nguyen, A. M., Schear, N., Jung, H., Godiyal, A., King, S. T., and Nguyen, H. D. (2009). Mavmm: Lightweight and purpose built vmm for malware analysis. In Computer Security Applications Conf., 2009. ACSAC ’09. Annual, pages 441–450.

Oleg, K. (2016). Anti-debug protection techniques: Implementation and neutralization. [link].

Pafish (2012). Pafish.

Paleari, R., Martignoni, L., Roglia, G. F., and Bruschi, D. (2009). A fistful of red-pills: How to automatically generate procedures to detect cpu emulators. In Proc. of the 3rd USENIX Conf. on Offensive Technologies,WOOT’09, pages 2–2, Berkeley, CA, USA. USENIX Association.

Peframe (2014). Peframe.

Pék, G., Bencsáth, B., and Buttyán, L. (2011). nether: In-guest detection of out-of-the-guest malware analyzers. In Proc. of the Fourth European Workshop on System Security, EUROSEC ’11, pages 3:1–3:6, New York, NY, USA. ACM.

Poulios, G. (2015). Ropinjector.

Poulios, G., Ntantogian, C., and Xenakis, C. (2015). Ropinjector: Using return oriented programming for polymorphism and antivirus evasion. [link].

Pyew (2012). Pyew.

Radare (2008). Radare. Access Date: May/2017.

Saleh, M., Ratazzi, E. P., and Xu, S. (2014). Instructions-based detection of sophisticated obfuscation and packing. In 2014 IEEE Military Communications Conf., pages 1–6.

Schwarz, B., Debray, S., and Andrews, G. (2002). Disassembly of executable code revisited. In Proc. of the NinthWorking Conf. on Reverse Engineering (WCRE’02), WCRE ’02, pages 45–,Washington, DC, USA. IEEE Computer Society. Securiteam (2004). Red pill... or how to detect vmm using (almost) one cpu instruction.

Sikorski, M. and Honig, A. (2012). Practical Malware Analysis: The Hands-On Guide to Dissecting Malicious Software. No Starch Press, San Francisco, CA, USA, 1st edition.

Singh, A. (2014). Not so fast my friend - using inverted timing attacks to bypass dynamic analysis. [link].

Smith, A. J., Mills, R. F., Bryant, A. R., Peterson, G. L., and Grimaila, M. R. (2014). Redir: Automated static detection of obfuscated anti-debugging techniques. In 2014 International Conf. on Collaboration Technologies and Systems (CTS), pages 173–180.

Vasudevan, A. and Yerraballi, R. (2006). Cobra: fine-grained malware analysis using stealth localized-executions. In 2006 IEEE Symposium on Security and Privacy (S P’06), pages 15 pp.–279.

VMWare (2010). Mechanisms to determine if software is running in a vmware virtual machine (1009458). [link].

Willems, C., Hund, R., Fobian, A., Felsch, D., Holz, T., and Vasudevan, A. (2012). Down to the bare metal: Using processor features for binary analysis. In Proc. of the 28th Annual Computer Security Applications Conf., ACSAC ’12, pages 189–198, New York, NY, USA. ACM.

Xiao, X., s. Zhang, X., and d. Li, X. (2010). New approach to path explosion problem of symbolic execution. In Pervasive Computing Signal Processing and Applications (PCSPA), 2010 First International Conf. on, pages 301–304.

Yokoyama, A., Ishii, K., Tanabe, R., Papa, Y., Yoshioka, K., Matsumoto, T., Kasama, T., Inoue, D., Brengel, M., Backes, M., and Rossow, C. (2016). SandPrint: Fingerprinting Malware Sandboxes to Provide Intelligence for Sandbox Evasion, pages 165–187. Springer International Publishing, Cham.
BOTACIN, Marcus; ROCHA, Vitor Falcão da; GEUS, Paulo Lício de; GRÉGIO, André. Analysis, Anti-Analysis, Anti-Anti-Analysis: An Overview of the Evasive Malware Scenario. In: SIMPÓSIO BRASILEIRO DE SEGURANÇA DA INFORMAÇÃO E DE SISTEMAS COMPUTACIONAIS (SBSEG), 17. , 2017, Brasília. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2017 . p. 250-263. DOI: