TVTAT - A Real Time Camera Imaging Testing Tool for Smart TVs: Preliminary Results
Resumo
Test automation tools that can accurately control a smart TV device are rare to be found. The difficulty of creating a system that is able to control such a device generates specific needs. In addition, the Brazilian digital television system supports DTVPlay, a middleware that provides the ability to broadcast interactive applications written in HTML5, NCL and Lua, that must be fully implemented on at least 90% of televisions manufactured in Brazil. Thus, there is a standard to be followed and a set of tests that need to be performed with each new middleware release. This work presents an automation tool called TV Test Automation Tool (TVTAT), that performs non-invasive tests on smart TVs, mainly but not restricted to DTVPlay tests. TVTAT uses real-time computer vision techniques such as optical character recognition, image pattern matching and color verification to assert that the middleware’s implementation is according to the published specification. The results of some test scenarios are presented, demonstrating that there are trends that can be found either in application performance situations or in tests of availability of transmitted applications that depend on DTVPlay.
Palavras-chave:
Testing Tools, Software Testing, Smart TV Testing, DTVPlay, Digital Television
Referências
Bestoun S. Ahmed and Miroslav Bures. 2019. Testing of smart TV applications: Key ingredients, challenges and proposed solutions. In Proceedings of the Future Technologies Conference (FTC) 2018: Volume 1. Springer, 241–256.
Bestoun S. Ahmed, Angelo Gargantini, and Miroslav Bures. 2020. An Automated Testing Framework For Smart TV apps Based on Model Separation. In 2020 IEEE International Conference on Software Testing, Verification and ValidationWorkshops (ICSTW). IEEE, 62–73.
Gabriella Alves, Rennan Barbosa, Raoni Kulesza, and Guido L.S. Filho. 2015. A Software Testing Process for Ginga Products. In Applications and Usability of Interactive TV: Third Iberoamerican Conference, jAUTI 2014, and Third Workshop on Interactive Digital TV, Held as Part of Webmedia 2014, João Pessoa, PB, Brazil, November 18-21, 2014. Revised Selected Papers 3. Springer, 61–73.
Gleb Avdeyenko and Teodor Narytnik. 2021. Hardware and Software Complex for Digital Television Signals Generation and Research. In 2021 IEEE 8th International Conference on Problems of Infocommunications, Science and Technology (PIC S&T). IEEE, 7–12.
Mohammad Yusaf Azimi, Celal Cagin Elgun, Atil Firat, Ferhat Erata, and Cemal Yilmaz. 2023. AdapTV: A Model-Based Test Adaptation Approach for End-to-End User Interface Testing of Smart TVs. IEEE Access 11 (2023), 32095–32118.
Gabriel Baum and Luiz Fernando G. Soares. 2012. Ginga middleware and digital TV in Latin America. IT Professional 14, 4 (2012), 59–61.
Axel Bons, Beatriz Marín, Pekka Aho, and Tanja E.J. Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.
Rodrigo Braga, Volnei Klehm, Lauro Gama, Michael Mello, Taynara Paiva, Dina Nogueira, Ruddá Beltrão, Addison Lima, Leonardo Santana, Lois Nascimento, et al. 2019. NuGingaJS: a full portable ITU-T H. 761 Ginga middleware for DTV and IPTV. In Proceedings of the 25th Brazillian Symposium on Multimedia and the Web. 257–264.
Miroslav Bures, Miroslav Macik, Bestoun S Ahmed, Vaclav Rechtberger, and Pavel Slavik. 2020. Testing the usability and accessibility of smart tv applications using an automated model-based approach. IEEE transactions on consumer electronics 66, 2 (2020), 134–143.
Jing Cao, Xiaoqiang Liu, Hui Guo, Lizhi Cai, and Yun Hu. 2021. Test case generation for web application based on markov reward process. In Journal of Physics: Conference Series, Vol. 1792. IOP Publishing, 012039.
Khasim Vali Dudekula, Hussain Syed, Mohamed Iqbal Mahaboob Basha, Sudhakar Ilango Swamykan, Purna Prakash Kasaraneni, Yellapragada Venkata Pavan Kumar, Aymen Flah, and Ahmad Taher Azar. 2023. Convolutional Neural Network-Based Personalized Program Recommendation System for Smart Television Users. Sustainability 15, 3 (2023), 2206.
Bruno Farias, Ivo Machado, Eddie B de Lima Filho, Cláudio Pinheiro, Petrina Kimura, Leonardo Cordeiro, and Daniel Xavier. 2022. A Methodology for Emulating, Developing, and Testing the Middleware DTV Play in Personal Computers. In 2022 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 1–6.
Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189–1232.
Fórum SBTVD [n. d.]. Normas Técnicas Sistema Brasileiro de TV Digital Terrestre. [link]. Accessed: 2023-05-25.
Ginga website [n. d.]. Ginga Kernel Description. [link]. Accessed: 2023-05-25.
Itseez. 2015. Open Source Computer Vision Library. [link].
Hyelynn Jeon and Sejong Oh. 2020. Hybrid-recursive feature elimination for efficient feature selection. Applied Sciences 10, 9 (2020), 3211.
Mihajlo Katona, Ivan Kastelan, Vukota Pekovic, Nikola Teslic, and Tarkan Tekcan. 2011. Automatic black box testing of television systems on the final production line. IEEE Transactions on Consumer Electronics 57, 1 (2011), 224–231.
Mumtaz Khan, Shah Khusro, Iftikhar Alam, Shaukat Ali, Inayat Khan, et al. 2022. Perspectives on the design, challenges, and evaluation of smart TV user interfaces. Scientific Programming 2022 (2022).
Orlewilson B. Maia, Andre R. da Silva Conceição, Manoel J. de Souza Júnior, Fabrício Izumi, Eddie B. de Lima Filho, and Paulo Corrêa. 2022. A Real-Time Analyzer for Testing DTV Play. In 2022 IEEE International Conference on Consumer Electronics (ICCE). 01–05. DOI: 10.1109/ICCE53296.2022.9730583
Dusica Marijan, Vladimir Zlokolica, Nikola Teslic, Vukota Pekovic, and Tarkan Tekcan. 2010. Automatic functional TV set failure detection system. IEEE Transactions on Consumer Electronics 56, 1 (2010), 125–133.
Owain Parry, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2021. A Survey of Flaky Tests. ACM Trans. Softw. Eng. Methodol. 31, 1, Article 17 (oct 2021), 74 pages. DOI: 10.1145/3476105
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825–2830.
Jônatas Rech, Vinícius Freitas, Bruno Farias, Eddie B de Lima Filho, Jeferson Costa, Ivo Machado, Xianpan Chen, Cláudio Pinheiro, and Daniel Xavier. 2021. A methodology for providing encrypted-content decoding in dtv play. In 2021 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 1–6.
Ray Smith. 2007. An overview of the Tesseract OCR engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), Vol. 2. IEEE, 629–633.
Greg Taylor. 2017. python-colormath Documentation. (2017).
Tiago H Trojahn, Juliano L Goncalves, Julio CB Mattos, Luciano V Agostini, and Leomar S Rosa. 2011. Tests and performance analysis of media processing implementations for the middleware of Brazilian Digital TV system using different scenarios. In 2011 Fifth FTRA International Conference on Multimedia and Ubiquitous Engineering. IEEE, 95–100.
Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 183–192.
Bestoun S. Ahmed, Angelo Gargantini, and Miroslav Bures. 2020. An Automated Testing Framework For Smart TV apps Based on Model Separation. In 2020 IEEE International Conference on Software Testing, Verification and ValidationWorkshops (ICSTW). IEEE, 62–73.
Gabriella Alves, Rennan Barbosa, Raoni Kulesza, and Guido L.S. Filho. 2015. A Software Testing Process for Ginga Products. In Applications and Usability of Interactive TV: Third Iberoamerican Conference, jAUTI 2014, and Third Workshop on Interactive Digital TV, Held as Part of Webmedia 2014, João Pessoa, PB, Brazil, November 18-21, 2014. Revised Selected Papers 3. Springer, 61–73.
Gleb Avdeyenko and Teodor Narytnik. 2021. Hardware and Software Complex for Digital Television Signals Generation and Research. In 2021 IEEE 8th International Conference on Problems of Infocommunications, Science and Technology (PIC S&T). IEEE, 7–12.
Mohammad Yusaf Azimi, Celal Cagin Elgun, Atil Firat, Ferhat Erata, and Cemal Yilmaz. 2023. AdapTV: A Model-Based Test Adaptation Approach for End-to-End User Interface Testing of Smart TVs. IEEE Access 11 (2023), 32095–32118.
Gabriel Baum and Luiz Fernando G. Soares. 2012. Ginga middleware and digital TV in Latin America. IT Professional 14, 4 (2012), 59–61.
Axel Bons, Beatriz Marín, Pekka Aho, and Tanja E.J. Vos. 2023. Scripted and scriptless GUI testing for web applications: An industrial case. Information and Software Technology 158 (2023), 107172.
Rodrigo Braga, Volnei Klehm, Lauro Gama, Michael Mello, Taynara Paiva, Dina Nogueira, Ruddá Beltrão, Addison Lima, Leonardo Santana, Lois Nascimento, et al. 2019. NuGingaJS: a full portable ITU-T H. 761 Ginga middleware for DTV and IPTV. In Proceedings of the 25th Brazillian Symposium on Multimedia and the Web. 257–264.
Miroslav Bures, Miroslav Macik, Bestoun S Ahmed, Vaclav Rechtberger, and Pavel Slavik. 2020. Testing the usability and accessibility of smart tv applications using an automated model-based approach. IEEE transactions on consumer electronics 66, 2 (2020), 134–143.
Jing Cao, Xiaoqiang Liu, Hui Guo, Lizhi Cai, and Yun Hu. 2021. Test case generation for web application based on markov reward process. In Journal of Physics: Conference Series, Vol. 1792. IOP Publishing, 012039.
Khasim Vali Dudekula, Hussain Syed, Mohamed Iqbal Mahaboob Basha, Sudhakar Ilango Swamykan, Purna Prakash Kasaraneni, Yellapragada Venkata Pavan Kumar, Aymen Flah, and Ahmad Taher Azar. 2023. Convolutional Neural Network-Based Personalized Program Recommendation System for Smart Television Users. Sustainability 15, 3 (2023), 2206.
Bruno Farias, Ivo Machado, Eddie B de Lima Filho, Cláudio Pinheiro, Petrina Kimura, Leonardo Cordeiro, and Daniel Xavier. 2022. A Methodology for Emulating, Developing, and Testing the Middleware DTV Play in Personal Computers. In 2022 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 1–6.
Jerome H Friedman. 2001. Greedy function approximation: a gradient boosting machine. Annals of statistics (2001), 1189–1232.
Fórum SBTVD [n. d.]. Normas Técnicas Sistema Brasileiro de TV Digital Terrestre. [link]. Accessed: 2023-05-25.
Ginga website [n. d.]. Ginga Kernel Description. [link]. Accessed: 2023-05-25.
Itseez. 2015. Open Source Computer Vision Library. [link].
Hyelynn Jeon and Sejong Oh. 2020. Hybrid-recursive feature elimination for efficient feature selection. Applied Sciences 10, 9 (2020), 3211.
Mihajlo Katona, Ivan Kastelan, Vukota Pekovic, Nikola Teslic, and Tarkan Tekcan. 2011. Automatic black box testing of television systems on the final production line. IEEE Transactions on Consumer Electronics 57, 1 (2011), 224–231.
Mumtaz Khan, Shah Khusro, Iftikhar Alam, Shaukat Ali, Inayat Khan, et al. 2022. Perspectives on the design, challenges, and evaluation of smart TV user interfaces. Scientific Programming 2022 (2022).
Orlewilson B. Maia, Andre R. da Silva Conceição, Manoel J. de Souza Júnior, Fabrício Izumi, Eddie B. de Lima Filho, and Paulo Corrêa. 2022. A Real-Time Analyzer for Testing DTV Play. In 2022 IEEE International Conference on Consumer Electronics (ICCE). 01–05. DOI: 10.1109/ICCE53296.2022.9730583
Dusica Marijan, Vladimir Zlokolica, Nikola Teslic, Vukota Pekovic, and Tarkan Tekcan. 2010. Automatic functional TV set failure detection system. IEEE Transactions on Consumer Electronics 56, 1 (2010), 125–133.
Owain Parry, Gregory M. Kapfhammer, Michael Hilton, and Phil McMinn. 2021. A Survey of Flaky Tests. ACM Trans. Softw. Eng. Methodol. 31, 1, Article 17 (oct 2021), 74 pages. DOI: 10.1145/3476105
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. 2011. Scikit-learn: Machine learning in Python. the Journal of machine Learning research 12 (2011), 2825–2830.
Jônatas Rech, Vinícius Freitas, Bruno Farias, Eddie B de Lima Filho, Jeferson Costa, Ivo Machado, Xianpan Chen, Cláudio Pinheiro, and Daniel Xavier. 2021. A methodology for providing encrypted-content decoding in dtv play. In 2021 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 1–6.
Ray Smith. 2007. An overview of the Tesseract OCR engine. In Ninth international conference on document analysis and recognition (ICDAR 2007), Vol. 2. IEEE, 629–633.
Greg Taylor. 2017. python-colormath Documentation. (2017).
Tiago H Trojahn, Juliano L Goncalves, Julio CB Mattos, Luciano V Agostini, and Leomar S Rosa. 2011. Tests and performance analysis of media processing implementations for the middleware of Brazilian Digital TV system using different scenarios. In 2011 Fifth FTRA International Conference on Multimedia and Ubiquitous Engineering. IEEE, 95–100.
Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 183–192.
Publicado
30/09/2024
Como Citar
CUNHA FILHO, Carlos Alberto Hagge da; MENDES, Hugo Abreu; PAULA, Adriano Rodrigues de; FIGUEIREDO, Ravi Barreto Doria; AZEVEDO, Jessamine Maria de Lima.
TVTAT - A Real Time Camera Imaging Testing Tool for Smart TVs: Preliminary Results. In: SIMPÓSIO BRASILEIRO DE TESTES DE SOFTWARE SISTEMÁTICO E AUTOMATIZADO (SAST), 9. , 2024, Curitiba/PR.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 1-10.
