Gaps in Operator Fusion for Deep Learning Systems: The Trade-off Between Compilation and Execution of Unfused Operations
Resumo
Operator fusion can reduce steady-state latency, but compiler work becomes relevant in short-lived inference workloads. We characterize native TVM, XLA, and PyTorch 2 (TorchInductor) stacks on ResNet-18/50, BERT, and GPT-2 using 𝐾 = 5 cold processes per model and input, 10 warmups, and 50 timed executions per process. We use the amortized cost model 𝑇 (𝑛) = 𝑇𝐶 + 𝑛 𝑇𝑋, where 𝑇𝐶 is the backend-specific compilation-cost proxy, 𝑇𝑋 the steady-state latency, and 𝑛 the number of executions, to expose the regimes favored by each measured stack. For TorchInductor, an inference-only Conv–BatchNorm parameter fold applied as offline preprocessing reduced the mean compiler-cost proxy by 29.5–41.1% for ResNet-18 and 1.9–28.6% for ResNet-50; in nine of the ten inputs the 95% confidence intervals remain non-overlapping after adding the measured 100–120-ms preprocessing cost. Execution effects were smaller and model-dependent. An applicability audit of the BERT LayerNorm–Linear fold finds no legal instance in the evaluated post-LayerNorm encoder: the structural matcher performs zero rewrites, and an isolated cold-process control shows overlapping confidence intervals in every input. The benchmark reports measured system-level trade-offs, while neither the native-stack comparison nor the valid ResNet fold establishes a universal compiler pass ordering.
Referências
Xuyi Cai, Ying Wang, and Lei Zhang. 2022. Optimus: An Operator Fusion Framework for Deep Neural Networks. ACM Transactions on Embedded Computing Systems 22, 1 (2022), 26 pages. DOI: 10.1145/3520142
Michael Canesche, Vanderson Martins do Rosario, Edson Borin, and Fernando Magno Quintão Pereira. 2025. Fusion of Operators of Computational Graphs via Greedy Clustering: The XNNC Experience. In Proceedings of the 34th ACM SIGPLAN International Conference on Compiler Construction. ACM, 117–127. DOI: 10.1145/3708493.3712689
Tianqi Chen, Thierry Moreau, Ziheng Jiang, Lianmin Zheng, Eddie Yan, Haichen Shen, Meghan Cowan, Leyuan Wang, Yuwei Hu, Luis Ceze, Carlos Guestrin, and Arvind Krishnamurthy. 2018. TVM: An Automated End-to-End Optimizing Compiler for Deep Learning. In 13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). 578–594.
Alain Darte. 1999. On the complexity of loop fusion. In Proceedings of the 1999 International Conference on Parallel Architectures and Compilation Techniques. IEEE, 149–157. Cat. No. PR00425.
Christopher W. Fraser. 1979. A Compact, Machine-Independent Peephole Optimizer. In Proceedings of the 6th ACM SIGACT-SIGPLAN Symposium on Principles of Programming Languages (POPL). ACM, 1–6. DOI: 10.1145/567752.567753
Chris Leary and Todd Wang. 2017. XLA: TensorFlow, Compiled. In TensorFlow Dev Summit. [link]
Wei Niu, Jiexiong Guan, Yanzhi Wang, Gagan Agrawal, and Bin Ren. 2021. DNNFusion: Accelerating Deep Neural Networks Execution with Advanced Operator Fusion. In Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation. ACM, 883–898. DOI: 10.1145/3453483.3454083
Suchita Pati, Shaizeen Aga, Nuwan Jayasena, and Matthew D. Sinclair. 2021. Demystifying BERT: Implications for Accelerator Design. In IEEE International Symposium on Workload Characterization (IISWC 2021). IEEE, 109–120. DOI: 10.1109/IISWC53511.2021.00019
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language Models are Unsupervised Multitask Learners. OpenAI Technical Report (2019).
Daniel Snider and Ruofan Liang. 2023. Operator Fusion in XLA: Analysis and Evaluation. [link]. arXiv:2301.13062.
Michael E.Wolf and Monica S. Lam. 1991. A Data Locality Optimizing Algorithm. In ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI). 30–44.
