The Importance of Ablation Studies for Complex Nonparametric Causal Models
Resumo
Ablation studies are essential for understanding the contribution of individual components within complex models, yet their application in nonparametric treatment effect estimation remains limited. This paper emphasizes the importance of ablation studies by examining the Bayesian Causal Forest (BCF) model, particularly the inclusion of the estimated propensity score π̂(xi) intended to mitigate regularization-induced confounding (RIC). Through an ablation study and three different conditional ablation studies, we demonstrate that excluding π̂(xi) does not diminish the model’s performance in estimating average and conditional average treatment effects or in uncertainty quantification. Moreover, omitting π̂(xi) reduces computational time by approximately 21%. These findings could suggest that the BCF model’s inherent flexibility suffices in adjusting for confounding without explicitly incorporating the propensity score. The study advocates for the routine use of ablation studies in treatment effect estimation to ensure model components are essential and to prevent unnecessary complexity.
Palavras-chave:
Conditional Average Treatment Effect, Average Treatment Effect, Bayesian Additive Regression Trees, Bayesian Causal Forest Model, Continuous Treatment Effect
Referências
Adhikari, A., Ram, A., Tang, R., and Lin, J. (2019). Rethinking complex neural network architectures for document classification. In Burstein, J., Doran, C., and Solorio, T., editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4046–4051, Minneapolis, Minnesota. Association for Computational Linguistics.
Bail, C. A., Guay, B., Maloney, E., Combs, A., Hillygus, D. S., Merhout, F., Freelon, D., and Volfovsky, A. (2019). Assessing the russian internet research agency’s impact on the political attitudes and behaviors of american twitter users in late 2017. Proceedings of the National Academy of Sciences, 117(1):243–250.
Ballinari, D. and Bearth, N. (2024). Improving the finite sample performance of double/debiased machine learning with propensity score calibration.
Cao, D., Wang, Y., Duan, J., Zhang, C., Zhu, X., Huang, C., Tong, Y., Xu, B., Bai, J., Tong, J., and Zhang, Q. (2020). Spectral temporal graph neural network for multivariate time-series forecasting. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 17766–17778. Curran Associates, Inc.
Chipman, H. A., George, E. I., and McCulloch, R. E. (2010). Bart: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1).
Du, L. (2020). How much deep learning does neural style transfer really need? an ablation study. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3150–3159. IEEE.
Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Hahn, P. R., Carvalho, C. M., Puelz, D., and He, J. (2018). Regularization and confounding in linear regression for treatment effect estimation. Bayesian Analysis, 13(1).
Hahn, P. R., Murray, J. S., and Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3).
Hasanpour, S. H., Rouhani, M., Fayyaz, M., and Sabokrou, M. (2016). Let’s keep it simple, using simple architectures to outperform deeper and more complex architectures.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217–240.
Li, F., Ding, P., and Mealli, F. (2023). Bayesian causal inference: a critical review. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381(2247).
Meyes, R., Lu, M., de Puiseau, C. W., and Meisen, T. (2019). Ablation studies in artificial neural networks.
Olivares, K. G., Challu, C., Marcjasz, G., Weron, R., and Dubrawski, A. (2023). Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with nbeatsx. International Journal of Forecasting, 39(2):884–900.
Papadogeorgou, G. and Li, F. (2020). Discussion for ‘bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects’. Bayesian Analysis, 15:1007–1013.
Prado, E. a. B., O’Neill, E., Hernández, B., Parnell, A. C., and Moral, R. A. (2020). Contributed discussion of Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects. Bayesian Analysis, 15(3):1029–1031. Discussion contribution to Hahn, Murray, and Carvalho.
Ročková, V. and Saha, E. (2019). On theory for bart. In Chaudhuri, K. and Sugiyama, M., editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 2839–2848. PMLR.
Ročková, V. and van der Pas, S. (2020). Posterior concentration for bayesian regression trees and forests. The Annals of Statistics, 48(4).
Shalit, U., Johansson, F. D., and Sontag, D. (2017). Estimating individual treatment effect: generalization bounds and algorithms. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3076–3085. PMLR.
Sheikholeslami, S., Meister, M., Wang, T., Payberah, A. H., Vlassov, V., and Dowling, J. (2021). Autoablation: Automated parallel ablation studies for deep learning. In Proceedings of the 1st Workshop on Machine Learning and Systems, EuroSys ’21. ACM.
Souto, H. G. and Louzada, F. (2024a). Advancing causal inference: A nonparametric approach to ATE and CATE estimation with continuous treatments.
Souto, H. G. and Louzada, F. (2024b). Beyond arbitrary replications: A principled approach to simulation design in causal inference.
Souto, H. G. and Moradi, A. (2024). Can transformers transform financial forecasting? China Finance Review International, 14(4):1–24.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228–1242.
WELCH, B. L. (1947). The generalization of ‘student’s’ problem when several different population varlances are involved. Biometrika, 34(1–2):28–35.
Wu, X., Xia, Y., Zhu, J., Wu, L., Xie, S., and Qin, T. (2022). A study of bert for context-aware neural machine translation. Machine Learning, 111(3):917–935.
Yeager, D. S., Bryan, C. J., Gross, J. J., Murray, J. S., Krettek Cobb, D., H. F. Santos, P., Gravelding, H., Johnson, M., and Jamieson, J. P. (2022). A synergistic mindsets intervention protects adolescents from stress. Nature, 607(7919):512–520.
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Yang, S. M., Carvalho, C. M., Hahn, P. R., Gopalan, M., Mhatre, P., Ferguson, R., Duckworth, A. L., and Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774):364–369.
Zigler, C. M., Watts, K., Yeh, R. W., Wang, Y., Coull, B. A., and Dominici, F. (2013). Model feedback in bayesian propensity score estimation. Biometrics, 69(1):263–273.
Bail, C. A., Guay, B., Maloney, E., Combs, A., Hillygus, D. S., Merhout, F., Freelon, D., and Volfovsky, A. (2019). Assessing the russian internet research agency’s impact on the political attitudes and behaviors of american twitter users in late 2017. Proceedings of the National Academy of Sciences, 117(1):243–250.
Ballinari, D. and Bearth, N. (2024). Improving the finite sample performance of double/debiased machine learning with propensity score calibration.
Cao, D., Wang, Y., Duan, J., Zhang, C., Zhu, X., Huang, C., Tong, Y., Xu, B., Bai, J., Tong, J., and Zhang, Q. (2020). Spectral temporal graph neural network for multivariate time-series forecasting. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H., editors, Advances in Neural Information Processing Systems, volume 33, pages 17766–17778. Curran Associates, Inc.
Chipman, H. A., George, E. I., and McCulloch, R. E. (2010). Bart: Bayesian additive regression trees. The Annals of Applied Statistics, 4(1).
Du, L. (2020). How much deep learning does neural style transfer really need? an ablation study. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3150–3159. IEEE.
Gomez, A. N., Ren, M., Urtasun, R., and Grosse, R. B. (2017). The reversible residual network: Backpropagation without storing activations. In Guyon, I., Luxburg, U. V., Bengio, S., Wallach, H., Fergus, R., Vishwanathan, S., and Garnett, R., editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc.
Hahn, P. R., Carvalho, C. M., Puelz, D., and He, J. (2018). Regularization and confounding in linear regression for treatment effect estimation. Bayesian Analysis, 13(1).
Hahn, P. R., Murray, J. S., and Carvalho, C. M. (2020). Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion). Bayesian Analysis, 15(3).
Hasanpour, S. H., Rouhani, M., Fayyaz, M., and Sabokrou, M. (2016). Let’s keep it simple, using simple architectures to outperform deeper and more complex architectures.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Hill, J. L. (2011). Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217–240.
Li, F., Ding, P., and Mealli, F. (2023). Bayesian causal inference: a critical review. Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 381(2247).
Meyes, R., Lu, M., de Puiseau, C. W., and Meisen, T. (2019). Ablation studies in artificial neural networks.
Olivares, K. G., Challu, C., Marcjasz, G., Weron, R., and Dubrawski, A. (2023). Neural basis expansion analysis with exogenous variables: Forecasting electricity prices with nbeatsx. International Journal of Forecasting, 39(2):884–900.
Papadogeorgou, G. and Li, F. (2020). Discussion for ‘bayesian regression tree models for causal inference: regularization, confounding, and heterogeneous effects’. Bayesian Analysis, 15:1007–1013.
Prado, E. a. B., O’Neill, E., Hernández, B., Parnell, A. C., and Moral, R. A. (2020). Contributed discussion of Bayesian Regression Tree Models for Causal Inference: Regularization, Confounding, and Heterogeneous Effects. Bayesian Analysis, 15(3):1029–1031. Discussion contribution to Hahn, Murray, and Carvalho.
Ročková, V. and Saha, E. (2019). On theory for bart. In Chaudhuri, K. and Sugiyama, M., editors, Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, volume 89 of Proceedings of Machine Learning Research, pages 2839–2848. PMLR.
Ročková, V. and van der Pas, S. (2020). Posterior concentration for bayesian regression trees and forests. The Annals of Statistics, 48(4).
Shalit, U., Johansson, F. D., and Sontag, D. (2017). Estimating individual treatment effect: generalization bounds and algorithms. In Precup, D. and Teh, Y. W., editors, Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 3076–3085. PMLR.
Sheikholeslami, S., Meister, M., Wang, T., Payberah, A. H., Vlassov, V., and Dowling, J. (2021). Autoablation: Automated parallel ablation studies for deep learning. In Proceedings of the 1st Workshop on Machine Learning and Systems, EuroSys ’21. ACM.
Souto, H. G. and Louzada, F. (2024a). Advancing causal inference: A nonparametric approach to ATE and CATE estimation with continuous treatments.
Souto, H. G. and Louzada, F. (2024b). Beyond arbitrary replications: A principled approach to simulation design in causal inference.
Souto, H. G. and Moradi, A. (2024). Can transformers transform financial forecasting? China Finance Review International, 14(4):1–24.
Szegedy, C., Liu, W., Jia, Y., Sermanet, P., Reed, S., Anguelov, D., Erhan, D., Vanhoucke, V., and Rabinovich, A. (2015). Going deeper with convolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
Wager, S. and Athey, S. (2018). Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228–1242.
WELCH, B. L. (1947). The generalization of ‘student’s’ problem when several different population varlances are involved. Biometrika, 34(1–2):28–35.
Wu, X., Xia, Y., Zhu, J., Wu, L., Xie, S., and Qin, T. (2022). A study of bert for context-aware neural machine translation. Machine Learning, 111(3):917–935.
Yeager, D. S., Bryan, C. J., Gross, J. J., Murray, J. S., Krettek Cobb, D., H. F. Santos, P., Gravelding, H., Johnson, M., and Jamieson, J. P. (2022). A synergistic mindsets intervention protects adolescents from stress. Nature, 607(7919):512–520.
Yeager, D. S., Hanselman, P., Walton, G. M., Murray, J. S., Crosnoe, R., Muller, C., Tipton, E., Schneider, B., Hulleman, C. S., Hinojosa, C. P., Paunesku, D., Romero, C., Flint, K., Roberts, A., Trott, J., Iachan, R., Buontempo, J., Yang, S. M., Carvalho, C. M., Hahn, P. R., Gopalan, M., Mhatre, P., Ferguson, R., Duckworth, A. L., and Dweck, C. S. (2019). A national experiment reveals where a growth mindset improves achievement. Nature, 573(7774):364–369.
Zigler, C. M., Watts, K., Yeh, R. W., Wang, Y., Coull, B. A., and Dominici, F. (2013). Model feedback in bayesian propensity score estimation. Biometrics, 69(1):263–273.
Publicado
19/10/2026
Como Citar
SOUTO, Hugo Gobato; LOUZADA, Francisco.
The Importance of Ablation Studies for Complex Nonparametric Causal Models. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 23. , 2026, Cuiabá/MT.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2026
.
p. 181-192.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2026.29736.
