On the Malware Detection Problem: Challenges & Novel Approaches

  • Marcus Botacin UFPR
  • Paulo de Geus UNICAMP
  • André Grégio UFPR


Many solutions to detect malware have been proposed over time, but effective and efficient malware detection still remains an open problem. In this work, I take a look at some malware detection challenges and pitfalls to contribute towards increasing system’s malware detection capabilities. I propose a new approach to tackle malware research in a practical but still scientific manner and leverage this approach to investigate four issues: (i) the need for understanding context to allow proper detection of localized threats; (ii) the need for developing better metrics for AntiVirus (AV) evaluation; (iii) the feasibility of leveraging hardware-software collaboration for efficient AV implementation, and (iv) the need for predicting future threats to allow faster incident responses.


Beppler, T., Botacin, M., Ceschin, F. J. O., Oliveira, L. E. S., and Grégio, A. (2019). L(a)ying in (test)bed. In Information Security. Springer.

Botacin, M. (2019). Análise do malware ativo na internet brasileira: 4 anos depois. o que mudou? https://gtergts.nic.br/.

Botacin, M. (2021). Does your threat model consider country and culture? a case study of brazilian internet banking security to show that it should! In USENIX Enigma.

Botacin, M., Aghakhani, H., Ortolani, S., Kruegel, C., Vigna, G., Oliveira, D., Geus, P. L. D., and Grégio, A. (2021a). One size does not fit all: A longitudinal analysis of brazilian financial malware. ACM TOPS.

Botacin, M., Alves, M. Z., Oliveira, D., and Grégio, A. (2022a). Heaven: A hardware-enhanced antivirus engine to accelerate real-time, signature-based malware detection. Elsevier ESWA.

Botacin, M., Bert ao, G., de Geus, P., Grégio, A., Kruegel, C., and Vigna, G. (2020a). On the security of application installers and online software repositories. In DIMVA. Springer.

Botacin, M., Ceschin, F., de Geus, P., and Grégio, A. (2020b). We need to talk about antiviruses: Challenges & pitfalls of av evaluations. Computers & Security.

Botacin, M., Ceschin, F., Sun, R., Oliveira, D., and Grégio, A. (2021b). Challenges and pitfalls in malware research. Computers & Security, page 102287.

Botacin, M., de Geus, P. L., and Grégio, A. (2019). “vanilla” malware: vanishing antiviruses by interleaving layers and layers of attacks. Comp. Vir. and Hack. Tech.

Botacin, M., de Geus, P. L., and Grégio, A. (2020c). Leveraging branch traces to understand kernel internals from within. Comp. Vir. and Hack. Tech.

Botacin, M., Domingues, F. D., Ceschin, F., Machnicki, R., Zanata Alves, M. A., de Geus, P. L., and Grégio, A. (2021c). Antiviruses under the microscope: A hands-on perspective. Comp. & Sec.

Botacin, M., Galante, L., Ceschin, F., Santos, P. C., Carro, L., de Geus, P., Grégio, A., and Alves, M. A. Z. (2019). The av says: Your hardware definitions were updated! In ReCoSoC.

Botacin, M., Galante, L., de Geus, P., and Grégio, A. (2019a). Revenge is a dish served cold: Debug-oriented malware decompilation and reassembly. In ROOTS. ACM.

Botacin, M., Galhardo Moia, V. H., Ceschin, F., Amaral Henriques, M. A., and Grégio, A. (2021d). Understanding uses and misuses of similarity hashing functions for malware detection and family clustering in actual scenarios. FSI: Digital Investigation.

Botacin, M., Grégio, A., and Alves, M. A. Z. (2020d). Near-memory & in-memory detection of fileless malware. In MEMSYS. ACM.

Botacin, M., Kalysch, A., and Grégio, A. (2019b). The internet banking [in]security spiral: Past, present, and future of online banking protection mechanisms based on a brazilian case study. In ARES. ACM.

Botacin, M., Moreira, F. B., Navaux, P. O. A., Grégio, A., and Alves, M. A. Z. (2022b). Terminator: A secure coprocessor to accelerate real-time antiviruses using inspection breakpoints. ACM Trans. Priv. Secur., 25(2).

Botacin, M., Zanata, M., and Grégio, A. (2020e). The self modifying code (smc)-aware processor (sap): a security look on architectural impact and support. Journal of Comp. Virology (JCVHT).

Ceschin, F., Botacin, M., Gomes, H. M., Oliveira, L. S., and Grégio, A. (2019). Shallow security: On the creation of adversarial variants to evade machine learning-based malware detectors. In ROOTS. ACM.

Ceschin, F., Botacin, M., Lüders, G., Gomes, H. M., Oliveira, L., and Gregio, A. (2020). No need to teach new tricks to old malware: Winning an evasion challenge with xor-based adversarial samples. In ROOTS. ACM.

Cohen, F. (1984). Computer viruses - theory and experiments. [link].

Grégio, A. and Botacin, M. (2020). Integridade, confidencialidade, disponibilidade, ransomware. [link].

Intel (2020). Technologies for hardware assisted native malware detection. [link].

Raffa, G. (2021). Testing antivirus in linux: An investigation on the effectiveness of solutions available for desktop computers. [link].

Sun, R., Botacin, M., Sapountzis, N., Yuan, X., Bishop, M., Porter, D. E., Li, X., Gregio, A., and Oliveira, D. (2020). A praise for defensive programming: Leveraging uncertainty for effective malware mitigation. IEEE TDSC.
BOTACIN, Marcus; DE GEUS, Paulo; GRÉGIO, André. On the Malware Detection Problem: Challenges & Novel Approaches. In: CONCURSO DE TESES E DISSERTAÇÕES - SIMPÓSIO BRASILEIRO DE SEGURANÇA DA INFORMAÇÃO E DE SISTEMAS COMPUTACIONAIS (SBSEG), 22. , 2022, Santa Maria. Anais [...]. Porto Alegre: Sociedade Brasileira de Computação, 2022 . p. 25-32. DOI: https://doi.org/10.5753/sbseg_estendido.2022.224746.

Artigos mais lidos do(s) mesmo(s) autor(es)

<< < 1 2