Identification of DNA Coding Regions Using Transformers - Cruz, Gustavo H. F.; Pozo, Aurora T. R. (Anais do Symposium on Knowledge Discovery, Mining and Learning (KDMiLe); 2025: Anais do XIII Symposium on Knowledge Discovery, Mining and LearningDO - 10.5753/kdmile.2025.247575, 2025)
Identifying coding (exon) and non-coding (intron) regions in DNA sequences is fundamental to understanding gene expression and its implications for biological processes and genetic diseases. In this work, we investigate the application of Transformer-based architectures to the task of intron and exon classification, comparing three distinct models: GPT-2, BERT, and DNABERT. These models were selected to evaluate the impact of context modeling strategies—autoregressive, bidirectional, and k-mer-based—on genomic sequence analysis. Experiments were carried out on a curated dataset comprising 100000 training sequences and 30000 test sequences, using mutually exclusive samples to ensure robust evaluation. All models were fine-tuned under uniform conditions, with a fixed batch size of 32 and learning rate constraints, and executed three times with different seeds. The results show that BERT achieved the highest classification accuracy (0.9905), outperforming DNABERT (0.9569) and GPT-2 (0.9867). While DNABERT was the fastest to train due to its k-mer tokenization and lighter computational requirements, its limited capacity to model long-range dependencies impaired its performance. In contrast, GPT-2 demonstrated competitive accuracy but at a higher computational cost, reinforcing the trade-off between generative modeling power and efficiency. This study highlights the importance of context-aware attention mechanisms in genomic sequence modeling and confirms the viability of Transformer architectures—especially bidirectional models like BERT—for high-accuracy classification of intronic and exonic regions. Future work may benefit from exploring larger models, sequence representation alternatives, and training optimization techniques to further enhance performance in genomics applications.