Linear Combinations of Bounded versus Unbounded Activation Functions: A Comparative Study on Noise Robustness
Resumo
Dados tabulares reais sofrem ruído de medição e perda de atributos, mas a literatura de ativações treináveis otimiza acurácia em dados limpos, enquanto a de robustez trata apenas ativações fixas. Investigamos essa lacuna: a limitação do contradomínio das primitivas confere robustez ao ensemble? Nas 72 tarefas do OpenML-CC18, sob ruído Gaussiano, Uniforme e Masking em cinco intensidades, comparamos combinações lineares treináveis limitadas (C19, C8), não limitadas (C22), mistas (C41) e a ReLU. Sem ruído, C22 e C41 lideram; sob ruído o quadro se inverte e as limitadas prevalecem (d ≈ 0,5 sob masking). A mistura C41 não herda a vantagem: o benefício exige limitação uniforme de todas as primitivas.Referências
Apicella, A., Donnarumma, F., Isgrò, F., and Prevete, R. (2021). A survey on modern trainable activation functions. Neural Networks, 138:14–32.
Bhattacharya, A., Gole, S. D., Chaudhuri, S., Frery, A. C., and Banerjee, B. (2026). Arcgate: Adaptive arctangent gated activation. arXiv preprint arXiv:2605.14518.
Bischl, B., Casalicchio, G., Feurer, M., Gijsbers, P., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., and Vanschoren, J. (2021). Openml benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
de Moraes, R. F. and Evangelista, R. d. S. (2023). Heuristics to reduce linear combinations of activation functions to improve image classification. In 2023 36th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pages 169–174. IEEE.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1–30.
Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. (2022). Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503:92–108.
Elliott, D. L. (1993). A better activation function for artificial neural networks. Technical Report ISR T.R. 93-8, Institute for Systems Research, University of Maryland.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
Li, Z., Liu, F., Yang, W., Peng, S., and Zhou, J. (2022). A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. TNNLS, 33:6999–7019.
Liew, S. S., Khalil-Hani, M., and Bakhteri, R. (2016). Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing, 216:718–734.
Mishra, A., Chandra, P., Ghose, U., and Sodhi, S. S. (2017). Bi-modal derivative adaptive activation function sigmoidal feedforward artificial neural networks. Applied Soft Computing, 61:983–994.
Molina, A., Schramowski, P., and Kersting, K. (2020). Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations (ICLR).
Pauli, P., Gramlich, D., Berberich, J., Allgöwer, F., and Scherer, C. W. (2021). Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters, 6:121–126.
Ru, Q., Hoang, L.-H., Hanif, M. A., and Shafique, M. (2020). Ft-clipact: Resilience analysis of deep neural networks and improving their fault tolerance using clipped activation. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pages 214–224. IEEE.
Schorn, C., Geissler, F., Qutub, S., and Dhamasia, A. (2021). Towards a safety case for hardware fault tolerance in convolutional neural networks using activation range supervision. In International Conference on Computer Safety, Reliability, and Security (SAFECOMP). Springer.
Sütfeld, L. R., Brieger, F., Finger, H., Füllhase, S., and Pipa, G. (2018). Adaptive blending units: Trainable activation functions for deep neural networks. In Proceedings of the SAI Intelligent Systems Conference (IntelliSys). arXiv:1806.10064.
Xu, B., Huang, R., and Li, M. (2016). Revise saturated activation functions. arXiv preprint arXiv:1602.05980.
Bhattacharya, A., Gole, S. D., Chaudhuri, S., Frery, A. C., and Banerjee, B. (2026). Arcgate: Adaptive arctangent gated activation. arXiv preprint arXiv:2605.14518.
Bischl, B., Casalicchio, G., Feurer, M., Gijsbers, P., Hutter, F., Lang, M., Mantovani, R. G., van Rijn, J. N., and Vanschoren, J. (2021). Openml benchmarking suites. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks.
de Moraes, R. F. and Evangelista, R. d. S. (2023). Heuristics to reduce linear combinations of activation functions to improve image classification. In 2023 36th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI), pages 169–174. IEEE.
Demšar, J. (2006). Statistical comparisons of classifiers over multiple data sets. Journal of Machine Learning Research, 7:1–30.
Dubey, S. R., Singh, S. K., and Chaudhuri, B. B. (2022). Activation functions in deep learning: A comprehensive survey and benchmark. Neurocomputing, 503:92–108.
Elliott, D. L. (1993). A better activation function for artificial neural networks. Technical Report ISR T.R. 93-8, Institute for Systems Research, University of Maryland.
LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P. (1998). Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324.
Li, Z., Liu, F., Yang, W., Peng, S., and Zhou, J. (2022). A Survey of Convolutional Neural Networks: Analysis, Applications, and Prospects. TNNLS, 33:6999–7019.
Liew, S. S., Khalil-Hani, M., and Bakhteri, R. (2016). Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. Neurocomputing, 216:718–734.
Mishra, A., Chandra, P., Ghose, U., and Sodhi, S. S. (2017). Bi-modal derivative adaptive activation function sigmoidal feedforward artificial neural networks. Applied Soft Computing, 61:983–994.
Molina, A., Schramowski, P., and Kersting, K. (2020). Padé activation units: End-to-end learning of flexible activation functions in deep networks. In International Conference on Learning Representations (ICLR).
Pauli, P., Gramlich, D., Berberich, J., Allgöwer, F., and Scherer, C. W. (2021). Training robust neural networks using lipschitz bounds. IEEE Control Systems Letters, 6:121–126.
Ru, Q., Hoang, L.-H., Hanif, M. A., and Shafique, M. (2020). Ft-clipact: Resilience analysis of deep neural networks and improving their fault tolerance using clipped activation. In 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), pages 214–224. IEEE.
Schorn, C., Geissler, F., Qutub, S., and Dhamasia, A. (2021). Towards a safety case for hardware fault tolerance in convolutional neural networks using activation range supervision. In International Conference on Computer Safety, Reliability, and Security (SAFECOMP). Springer.
Sütfeld, L. R., Brieger, F., Finger, H., Füllhase, S., and Pipa, G. (2018). Adaptive blending units: Trainable activation functions for deep neural networks. In Proceedings of the SAI Intelligent Systems Conference (IntelliSys). arXiv:1806.10064.
Xu, B., Huang, R., and Li, M. (2016). Revise saturated activation functions. arXiv preprint arXiv:1602.05980.
Publicado
19/10/2026
Como Citar
MORAES, Rogério Ferreira de; SILVA, Natália Azeredo Viana da; CORTES, Nattan Gomes Silva.
Linear Combinations of Bounded versus Unbounded Activation Functions: A Comparative Study on Noise Robustness. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 397-408.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.29994.
