Comparing Computational Architectures for Automated Journalism
Resumo
The majority of NLG systems have been designed following either a template-based or a pipeline-based architecture. Recent neural models for datato-text generation have been proposed with an end-to-end deep learning flavor, which handles non-linguistic input in natural language without explicit intermediary representations. This study compares the most often employed methods for generating Brazilian Portuguese texts from structured data. Results suggest that explicit intermediate steps in the generation process produce better texts than the ones generated by neural end-to-end architectures, avoiding data hallucination while better generalizing to unseen inputs. Code and corpus are publicly available.
Palavras-chave:
Natural Language Generation, Automated Journalism, Blue Amazon
Referências
Campos, J., Teixeira, A., Ferreira, T., Cozman, F., and Pagano, A. (2020). Towards Fully Automated News Reporting in Brazilian Portuguese. In Anais do XVII Encontro Nacional de Inteligência Artificial e Computacional, pages 543-554. SBC.
Chen, X.-W. and Lin, X. (2014). Big data deep learning: challenges and perspectives. IEEE access, 2:514-525.
Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259.
Deemter, K. V., Theune, M., and Krahmer, E. (2005). Real versus template-based natural language generation: A false opposition? Computational linguistics, 31(1):15-24.
Dimitromanolaki, A. and Androutsopoulos, I. (2003). Learning to order facts for discourse planning in natural language generation. arXiv preprint cs/0306062.
Dušek, O., Novikova, J., and Rieser, V. (2018). Findings of the e2e nlg challenge. arXiv preprint arXiv:1810.01170.
e Costa, B. H., Gonçalves, J. M., and Gonçalves, E. J. (2022). Un Ocean Conference needs transparent and science-based leadership on ocean conservation. Marine Policy, 143:105197.
Ferreira, T. C., van der Lee, C., Van Miltenburg, E., and Krahmer, E. (2019). Neural data-to-text generation: A comparison between pipeline and end-to-end architectures.
Furtado, S. d. F. D. (2020). Automated journalism in brazil: an analysis of three robots on Twitter. Brazilian journalism research, 16(3):476-501.
Gatt, A. and Krahmer, E. (2018). Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65-170.
Graefe, A. (2016). Guide to automated journalism.
He, X. and Deng, L. (2018). Deep learning in natural language generation from images. In Deep learning in natural language processing, pages 289-307. Springer.
Heilbron, M., Ehinger, B., Hagoort, P., and De Lange, F. P. (2019). Tracking naturalistic linguistic predictions with deep neural language models. arXiv preprint arXiv:1909.04400.
Horacek, H. (2001). Building natural language generation systems.
Joshi, A., Kale, S., Chandel, S., and Pal, D. K. (2015). Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396.
Juraska, J. and Walker, M. (2021). Attention is indeed all you need: Semantically attention-guided decoding for data-to-text NLG. arXiv preprint arXiv:2109.07043.
Kim, T.-Y., Bae, S.-H., and An, Y.-E. (2020). Design of smart home implementation within IoT natural language interface. IEEE Access, 8:84929-84949.
Lavie, A. and Denkowski, M. J. (2009). The METEOR metric for automatic evaluation of machine translation. Machine translation, 23(2):105-115.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
Li, H. (2017). Deep learning for natural language processing: advantages and challenges. National Science Review.
Liddy, E. D. (2001). Natural language processing.
Lin, C.-Y. and Och, F. (2004). Looking for a few good metrics: ROUGE and its evaluation. In Ntcir workshop.
Mutton, A., Dras, M., Wan, S., and Dale, R. (2007). GLEU: Automatic evaluation of sentence-level fluency. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 344-351.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Pereira, J. C., Teixeira, A., and Pinto, J. S. (2015). Towards a hybrid NLG system for data2text in Portuguese. In 2015 10th Iberian Conference on Information Systems and Technologies (CISTI), pages 1-6. IEEE.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Reiter, E. (1995). Nlg vs. templates. arXiv preprint cmp-lg/9504013.
Reiter, E. and Dale, R. (2000). Building Applied Natural Language Generation Systems. Natural Language Engineering.
Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. (2022). BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188.
Smiley, C., Davoodi, E., Song, D., and Schilder, F. (2018). The E2E NLG challenge: A tale of two systems. In Proceedings of the 11th International Conference on Natural Language Generation, pages 472-477.
Sripada, S. G., Reiter, E., Davy, I., and Nilssen, K. (2004). Lessons from deploying NLG technology for marine weather forecast text generation. WEATHER, 5:7.
Stede, M. (1994). Lexicalization in natural language generation: A survey. Artificial Intelligence Review, 8(4):309-336.
Stonebraker, M. (2010). Sql databases v. nosql databases. Communications of the ACM, 53(4):10-11.
Teixeira, A. L. R., Campos, J., Cunha, R., Ferreira, T. C., Pagano, A., and Cozman, F. (2020). DaMata: A robot-journalist covering the Brazilian Amazon deforestation. In Proceedings of the 13th International Conference on Natural Language Generation.
Thompson, N. and Muggah, R. (2015). The Blue Amazon. Foreign affairs, 11.
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
Chen, X.-W. and Lin, X. (2014). Big data deep learning: challenges and perspectives. IEEE access, 2:514-525.
Cho, K., Van Merriënboer, B., Bahdanau, D., and Bengio, Y. (2014). On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprint arXiv:1409.1259.
Deemter, K. V., Theune, M., and Krahmer, E. (2005). Real versus template-based natural language generation: A false opposition? Computational linguistics, 31(1):15-24.
Dimitromanolaki, A. and Androutsopoulos, I. (2003). Learning to order facts for discourse planning in natural language generation. arXiv preprint cs/0306062.
Dušek, O., Novikova, J., and Rieser, V. (2018). Findings of the e2e nlg challenge. arXiv preprint arXiv:1810.01170.
e Costa, B. H., Gonçalves, J. M., and Gonçalves, E. J. (2022). Un Ocean Conference needs transparent and science-based leadership on ocean conservation. Marine Policy, 143:105197.
Ferreira, T. C., van der Lee, C., Van Miltenburg, E., and Krahmer, E. (2019). Neural data-to-text generation: A comparison between pipeline and end-to-end architectures.
Furtado, S. d. F. D. (2020). Automated journalism in brazil: an analysis of three robots on Twitter. Brazilian journalism research, 16(3):476-501.
Gatt, A. and Krahmer, E. (2018). Survey of the state of the art in natural language generation: Core tasks, applications and evaluation. Journal of Artificial Intelligence Research, 61:65-170.
Graefe, A. (2016). Guide to automated journalism.
He, X. and Deng, L. (2018). Deep learning in natural language generation from images. In Deep learning in natural language processing, pages 289-307. Springer.
Heilbron, M., Ehinger, B., Hagoort, P., and De Lange, F. P. (2019). Tracking naturalistic linguistic predictions with deep neural language models. arXiv preprint arXiv:1909.04400.
Horacek, H. (2001). Building natural language generation systems.
Joshi, A., Kale, S., Chandel, S., and Pal, D. K. (2015). Likert scale: Explored and explained. British journal of applied science & technology, 7(4):396.
Juraska, J. and Walker, M. (2021). Attention is indeed all you need: Semantically attention-guided decoding for data-to-text NLG. arXiv preprint arXiv:2109.07043.
Kim, T.-Y., Bae, S.-H., and An, Y.-E. (2020). Design of smart home implementation within IoT natural language interface. IEEE Access, 8:84929-84949.
Lavie, A. and Denkowski, M. J. (2009). The METEOR metric for automatic evaluation of machine translation. Machine translation, 23(2):105-115.
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoyanov, V., and Zettlemoyer, L. (2019). Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.
Li, H. (2017). Deep learning for natural language processing: advantages and challenges. National Science Review.
Liddy, E. D. (2001). Natural language processing.
Lin, C.-Y. and Och, F. (2004). Looking for a few good metrics: ROUGE and its evaluation. In Ntcir workshop.
Mutton, A., Dras, M., Wan, S., and Dale, R. (2007). GLEU: Automatic evaluation of sentence-level fluency. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 344-351.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J. (2002). Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311-318.
Pereira, J. C., Teixeira, A., and Pinto, J. S. (2015). Towards a hybrid NLG system for data2text in Portuguese. In 2015 10th Iberian Conference on Information Systems and Technologies (CISTI), pages 1-6. IEEE.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., et al. (2019). Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Reiter, E. (1995). Nlg vs. templates. arXiv preprint cmp-lg/9504013.
Reiter, E. and Dale, R. (2000). Building Applied Natural Language Generation Systems. Natural Language Engineering.
Shuster, K., Xu, J., Komeili, M., Ju, D., Smith, E. M., Roller, S., Ung, M., Chen, M., Arora, K., Lane, J., et al. (2022). BlenderBot 3: a deployed conversational agent that continually learns to responsibly engage. arXiv preprint arXiv:2208.03188.
Smiley, C., Davoodi, E., Song, D., and Schilder, F. (2018). The E2E NLG challenge: A tale of two systems. In Proceedings of the 11th International Conference on Natural Language Generation, pages 472-477.
Sripada, S. G., Reiter, E., Davy, I., and Nilssen, K. (2004). Lessons from deploying NLG technology for marine weather forecast text generation. WEATHER, 5:7.
Stede, M. (1994). Lexicalization in natural language generation: A survey. Artificial Intelligence Review, 8(4):309-336.
Stonebraker, M. (2010). Sql databases v. nosql databases. Communications of the ACM, 53(4):10-11.
Teixeira, A. L. R., Campos, J., Cunha, R., Ferreira, T. C., Pagano, A., and Cozman, F. (2020). DaMata: A robot-journalist covering the Brazilian Amazon deforestation. In Proceedings of the 13th International Conference on Natural Language Generation.
Thompson, N. and Muggah, R. (2015). The Blue Amazon. Foreign affairs, 11.
Xue, L., Constant, N., Roberts, A., Kale, M., Al-Rfou, R., Siddhant, A., Barua, A., and Raffel, C. (2020). mt5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934.
Publicado
28/11/2022
Como Citar
SYM, Yan V.; CAMPOS, João Gabriel M.; JOSÉ, Marcos M.; COZMAN, Fabio G..
Comparing Computational Architectures for Automated Journalism. In: ENCONTRO NACIONAL DE INTELIGÊNCIA ARTIFICIAL E COMPUTACIONAL (ENIAC), 19. , 2022, Campinas/SP.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2022
.
p. 377-388.
ISSN 2763-9061.
DOI: https://doi.org/10.5753/eniac.2022.227426.
