A Missing Data Imputation GAN for Character Sprite Generation
Resumo
Creating and updating pixel art character sprites with many frames spanning different animations and poses takes time and can quickly become repetitive. However, that can be partially automated to allow artists to focus on more creative tasks. In this work, we concentrate on creating pixel art character sprites in a target pose from images of them facing other three directions. We present a novel approach to character generation by framing the problem as a missing data imputation task. Our proposed generative adversarial networks model receives the images of a character in all available domains and produces the image of the missing pose. We evaluated our approach in the scenarios with one, two, and three missing images, achieving similar or better results to the state-of-the-art when more images are available. We also evaluate the impact of the proposed changes to the base architecture.
Palavras-chave:
Generative Adversarial Networks, Procedural Content Generation, Image-to-Image Translation, Missing Data Imputation, Character Sprites
Referências
Afifi, M., Brubaker, M. A., and Brown, M. S. (2021). HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histograms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7941–7950, Virtual. IEEE Computer Society.
Buzuti, L. F. and Thomaz, C. E. (2023). Fréchet AutoEncoder Distance: A new approach for evaluation of Generative Adversarial Networks. Computer Vision and Image Understanding, 235:1–11.
Choi, J.-I., Kim, S.-K., and Kang, S.-J. (2022). Image Translation Method for Game Character Sprite Drawing. Computer Modeling in Engineering & Sciences, 131(2):747–762.
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. (2018). StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8789–8797, Salt Lake City. IEEE.
Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. (2020). StarGAN v2: Diverse Image Synthesis for Multiple Domains. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8185–8194, Seattle. IEEE.
Coutinho, F. and Chaimowicz, L. (2022a). Generating Pixel Art Character Sprites using GANs. In 2022 21st Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pages 1–6, Natal, Brazil. IEEE.
Coutinho, F. and Chaimowicz, L. (2022b). On the Challenges of Generating Pixel Art Character Sprites Using GANs. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 18(1):87–94.
Coutinho, F. and Chaimowicz, L. (2024). Pixel art character generation as an image-to-image translation problem using GANs. Graphical Models, 132:101213.
Gonzalez, A., Guzdial, M., and Ramos, F. (2020). Generating Gameplay-Relevant Art Assets with Transfer Learning. In Proceedings of the AIIDE Workshop on Experimental AI in Games, pages 1–7, Worcester. ArXiV.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. In NIPS’14: Proceedings of the 27th International Conference on Neural Information Processing Systems, volume 29, pages 2672–2680, Cambridge. MIT Press.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, 2017-December:6627–6638.
Hong, S., Kim, S., and Kang, S. (2019). Game sprite generator using a multi discriminator GAN. KSII Transactions on Internet and Information Systems, 13(8):4255–4269.
Huang, X., Liu, M. Y., Belongie, S., and Kautz, J. (2018). Multimodal Unsupervised Image-to-Image Translation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11207 LNCS:179–196.
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. (2017). Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2017-Janua, pages 5967–5976, Honolulu. IEEE.
Jiang, Z. and Sweetser, P. (2021). GAN-Assisted YUV Pixel Art Generation. In Australasian Joint Conference on Artificial Intelligence, pages 1–12, Sydney. Springer International Publishing.
Lee, D., Kim, J., Moon, W.-J., and Ye, J. C. (2019). CollaGAN: Collaborative GAN for Missing Image Data Imputation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 2019-June, pages 2482–2491, Long Beach. IEEE.
Liu, J., Pasumarthi, S., Duffy, B., Gong, E., Datta, K., and Zaharchuk, G. (2023). One Model to Synthesize Them All: Multi-Contrast Multi-Scale Transformer for Missing Data Imputation. IEEE Transactions on Medical Imaging, 42(9):2577–2591.
Loftsdottir, D. and Guzdial, M. (2022). SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches. In Proceedings of the 17th International Conference on the Foundations of Digital Games, pages 1–7, New York. ACM.
Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Smolley, S. P. (2017). Least Squares Generative Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2813–2821, Venice. IEEE.
Moreira, R. D., Coutinho, F., and Chaimowicz, L. (2022). Analysis and Compilation of Normal Map Generation Techniques for Pixel Art. In 2022 21st Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pages 1–6, Natal. IEEE.
Pang, Y., Lin, J., Qin, T., and Chen, Z. (2022). Image-to-Image Translation: Methods and Applications. IEEE Transactions on Multimedia, 24:3859–3881.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022- June:10674–10685.
Saravanan, A. and Guzdial, M. (2022). Pixel VQ-VAEs for Improved Pixel Art Representation. In Experimental AI in Games Workshop (EXAG) 2022, pages 1–9, Pomona. ArXiv.
Schreier, J. (2017). Blood, Sweat, and Pixels: The Triumphant, Turbulent Stories Behind How Video Games Are Made. HarperCollins, New York.
Serpa, Y. R. and Rodrigues, M. A. F. (2019). Towards Machine-Learning Assisted Asset Generation for Games: A Study on Pixel Art Sprite Sheets. In 2019 18th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), volume 2019-Octob, pages 182–191, Rio de Janeiro. IEEE.
Serpa, Y. R. and Rodrigues, M. A. F. (2022). Human and machine collaboration for painting game assets with deep learning. Entertainment Computing, 43:100497.
Shang, C., Palmer, A., Sun, J., Chen, K.-S., Lu, J., and Bi, J. (2017). VIGAN: Missing view imputation with generative adversarial networks. In 2017 IEEE International Conference on Big Data (Big Data), pages 766–775, Boston. IEEE.
Sharma, A., Member, S., Hamarneh, G., and Member, S. (2019). Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network. IEEE Transactions on Medical Imaging, 39(4):1170–1183.
Shen, L., Zhu, W., Wang, X., Xing, L., Pauly, J. M., Turkbey, B., Harmon, S. A., Sanford, T. H., Mehralivand, S., Choyke, P. L., Wood, B. J., and Xu, D. (2021). Multi-Domain Image Completion for Random Missing Input Data. IEEE Transactions on Medical Imaging, 40(4):1113–1122.
Silber, D. (2015). Pixel Art for Game Developers. CRC Press, Boca Raton.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December:2818–2826.
van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel Recurrent Neural Networks Koray Kavukcuoglu. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, pages 1747–1756, New York, NY, USA. JMLR.org.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. (2017). Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, volume 30, pages 1–10, Long Beach. Curran Associates, Inc.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612.
Yoon, J., Jordon, J., and Van Der Schaar, M. (2018). GAIN: Missing Data Imputation using Generative Adversarial Nets. 35th International Conference on Machine Learning, ICML 2018, 13:9042–9051.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. (2017a). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242–2251, Venice. IEEE.
Zhu, J. Y., Zhang, R., Pathak, D., Darrell, T., Efros, A. A., Wang, O., and Shechtman, E. (2017b). Toward Multimodal Image-to-Image Translation. Advances in Neural Information Processing Systems, 2017-December:466–477.
Buzuti, L. F. and Thomaz, C. E. (2023). Fréchet AutoEncoder Distance: A new approach for evaluation of Generative Adversarial Networks. Computer Vision and Image Understanding, 235:1–11.
Choi, J.-I., Kim, S.-K., and Kang, S.-J. (2022). Image Translation Method for Game Character Sprite Drawing. Computer Modeling in Engineering & Sciences, 131(2):747–762.
Choi, Y., Choi, M., Kim, M., Ha, J.-W., Kim, S., and Choo, J. (2018). StarGAN: Unified Generative Adversarial Networks for Multi-domain Image-to-Image Translation. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8789–8797, Salt Lake City. IEEE.
Choi, Y., Uh, Y., Yoo, J., and Ha, J.-W. (2020). StarGAN v2: Diverse Image Synthesis for Multiple Domains. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8185–8194, Seattle. IEEE.
Coutinho, F. and Chaimowicz, L. (2022a). Generating Pixel Art Character Sprites using GANs. In 2022 21st Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pages 1–6, Natal, Brazil. IEEE.
Coutinho, F. and Chaimowicz, L. (2022b). On the Challenges of Generating Pixel Art Character Sprites Using GANs. Proceedings of the AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment, 18(1):87–94.
Coutinho, F. and Chaimowicz, L. (2024). Pixel art character generation as an image-to-image translation problem using GANs. Graphical Models, 132:101213.
Gonzalez, A., Guzdial, M., and Ramos, F. (2020). Generating Gameplay-Relevant Art Assets with Transfer Learning. In Proceedings of the AIIDE Workshop on Experimental AI in Games, pages 1–7, Worcester. ArXiV.
Goodfellow, I. J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y. (2014). Generative Adversarial Nets. In NIPS’14: Proceedings of the 27th International Conference on Neural Information Processing Systems, volume 29, pages 2672–2680, Cambridge. MIT Press.
Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., and Hochreiter, S. (2017). GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. Advances in Neural Information Processing Systems, 2017-December:6627–6638.
Hong, S., Kim, S., and Kang, S. (2019). Game sprite generator using a multi discriminator GAN. KSII Transactions on Internet and Information Systems, 13(8):4255–4269.
Huang, X., Liu, M. Y., Belongie, S., and Kautz, J. (2018). Multimodal Unsupervised Image-to-Image Translation. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11207 LNCS:179–196.
Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A. (2017). Image-to-Image Translation with Conditional Adversarial Networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), volume 2017-Janua, pages 5967–5976, Honolulu. IEEE.
Jiang, Z. and Sweetser, P. (2021). GAN-Assisted YUV Pixel Art Generation. In Australasian Joint Conference on Artificial Intelligence, pages 1–12, Sydney. Springer International Publishing.
Lee, D., Kim, J., Moon, W.-J., and Ye, J. C. (2019). CollaGAN: Collaborative GAN for Missing Image Data Imputation. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), volume 2019-June, pages 2482–2491, Long Beach. IEEE.
Liu, J., Pasumarthi, S., Duffy, B., Gong, E., Datta, K., and Zaharchuk, G. (2023). One Model to Synthesize Them All: Multi-Contrast Multi-Scale Transformer for Missing Data Imputation. IEEE Transactions on Medical Imaging, 42(9):2577–2591.
Loftsdottir, D. and Guzdial, M. (2022). SketchBetween: Video-to-Video Synthesis for Sprite Animation via Sketches. In Proceedings of the 17th International Conference on the Foundations of Digital Games, pages 1–7, New York. ACM.
Mao, X., Li, Q., Xie, H., Lau, R. Y., Wang, Z., and Smolley, S. P. (2017). Least Squares Generative Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2813–2821, Venice. IEEE.
Moreira, R. D., Coutinho, F., and Chaimowicz, L. (2022). Analysis and Compilation of Normal Map Generation Techniques for Pixel Art. In 2022 21st Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), pages 1–6, Natal. IEEE.
Pang, Y., Lin, J., Qin, T., and Chen, Z. (2022). Image-to-Image Translation: Methods and Applications. IEEE Transactions on Multimedia, 24:3859–3881.
Rombach, R., Blattmann, A., Lorenz, D., Esser, P., and Ommer, B. (2021). High-Resolution Image Synthesis with Latent Diffusion Models. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2022- June:10674–10685.
Saravanan, A. and Guzdial, M. (2022). Pixel VQ-VAEs for Improved Pixel Art Representation. In Experimental AI in Games Workshop (EXAG) 2022, pages 1–9, Pomona. ArXiv.
Schreier, J. (2017). Blood, Sweat, and Pixels: The Triumphant, Turbulent Stories Behind How Video Games Are Made. HarperCollins, New York.
Serpa, Y. R. and Rodrigues, M. A. F. (2019). Towards Machine-Learning Assisted Asset Generation for Games: A Study on Pixel Art Sprite Sheets. In 2019 18th Brazilian Symposium on Computer Games and Digital Entertainment (SBGames), volume 2019-Octob, pages 182–191, Rio de Janeiro. IEEE.
Serpa, Y. R. and Rodrigues, M. A. F. (2022). Human and machine collaboration for painting game assets with deep learning. Entertainment Computing, 43:100497.
Shang, C., Palmer, A., Sun, J., Chen, K.-S., Lu, J., and Bi, J. (2017). VIGAN: Missing view imputation with generative adversarial networks. In 2017 IEEE International Conference on Big Data (Big Data), pages 766–775, Boston. IEEE.
Sharma, A., Member, S., Hamarneh, G., and Member, S. (2019). Missing MRI Pulse Sequence Synthesis using Multi-Modal Generative Adversarial Network. IEEE Transactions on Medical Imaging, 39(4):1170–1183.
Shen, L., Zhu, W., Wang, X., Xing, L., Pauly, J. M., Turkbey, B., Harmon, S. A., Sanford, T. H., Mehralivand, S., Choyke, P. L., Wood, B. J., and Xu, D. (2021). Multi-Domain Image Completion for Random Missing Input Data. IEEE Transactions on Medical Imaging, 40(4):1113–1122.
Silber, D. (2015). Pixel Art for Game Developers. CRC Press, Boca Raton.
Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2015). Rethinking the Inception Architecture for Computer Vision. Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December:2818–2826.
van den Oord, A., Kalchbrenner, N., and Kavukcuoglu, K. (2016). Pixel Recurrent Neural Networks Koray Kavukcuoglu. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, pages 1747–1756, New York, NY, USA. JMLR.org.
van den Oord, A., Vinyals, O., and Kavukcuoglu, K. (2017). Neural Discrete Representation Learning. In Advances in Neural Information Processing Systems, volume 30, pages 1–10, Long Beach. Curran Associates, Inc.
Wang, Z., Bovik, A. C., Sheikh, H. R., and Simoncelli, E. P. (2004). Image quality assessment: From error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600–612.
Yoon, J., Jordon, J., and Van Der Schaar, M. (2018). GAIN: Missing Data Imputation using Generative Adversarial Nets. 35th International Conference on Machine Learning, ICML 2018, 13:9042–9051.
Zhu, J.-Y., Park, T., Isola, P., and Efros, A. A. (2017a). Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242–2251, Venice. IEEE.
Zhu, J. Y., Zhang, R., Pathak, D., Darrell, T., Efros, A. A., Wang, O., and Shechtman, E. (2017b). Toward Multimodal Image-to-Image Translation. Advances in Neural Information Processing Systems, 2017-December:466–477.
Publicado
30/09/2024
Como Citar
COUTINHO, Flávio; CHAIMOWICZ, Luiz.
A Missing Data Imputation GAN for Character Sprite Generation. In: SIMPÓSIO BRASILEIRO DE JOGOS E ENTRETENIMENTO DIGITAL (SBGAMES), 23. , 2024, Manaus/AM.
Anais [...].
Porto Alegre: Sociedade Brasileira de Computação,
2024
.
p. 436-455.
DOI: https://doi.org/10.5753/sbgames.2024.241116.
