Generative Artificial Intelligence in Software Engineering: Enhancing Code Generation, Testing, and Developer Productivity Suresh Babu Narra
Keywords:
Generative AI, large language models, code generation, code completion, pass@k, fuzz testing, test generation, developer productivity, memorization, software engineering.Abstract
This study examines the impact of generative synthetic intelligence (GenAI) on software program construction, testing, and maintenance; however, because proprietary models and non-reproducible environments limit reproducibility, insufficient evidence exists to pinpoint the conditions under which GenAI can have an influence. This study examines how generative AI (GenAI) influences software development, testing, and maintenance, but because proprietary models and environments cannot be reproduced, it is difficult to determine when and how GenAI exerts influence. This paper introduces a fully reproducible lab study in which we test models trained from scratch on generative code from 8.5 million characters of real Python standard-library code end-to-end across three software-engineering tasks: code generation, testing, and developer assistance. Attempts are made to compare how these models perform on held-out files, using character-level n-gram models and a 2.24-million-parameter neural language model, as well as a mixture of these. With the hybrid model, cross-entropy is 1.403 bits per character (19.1% less than the best n-gram model), and the fraction of correctly predicted next characters is 74.1%. The neural model completes 10.7% of held-out lines, compared with 0.3% for the 6-gram model. For perplexity, lower perplexity does not mean better generation; for sampling, likelihood-based sampling doubles exact-match at moderate temperature. The number of verbatim reproduced training code increases by a factor of 5 from the 4-gram to the 10-gram model (4.5% to 23.9% of 32-character windows at T = 0.5) and decreases as the temperature drops—only the largest n-gram has the overlap value human-written code has naturally with the corpus (23.9%). For three pure-Python parsers, a language-dedicated generator (GenAI sampling and mutation of actual Python code) is more effective than random inputs in the case of the Python unparser but not for parsers of other languages. When coverage feedback was added, the percentage of valid generated inputs also rose (from 2% to 65%), but the coverage decreased this time, demonstrating this trade-off between validity and diversity. An inline suggestion can save 2-32.1% of keystrokes, and we can tune it according to the acceptance rate and saving rate using a keystroke-level simulation. The results provide strong recommendations for using GenAI in software engineering, including the release of all code and data.
References
[1] M. Chen et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374, 2021. https://doi.org/10.48550/arXiv.2107.03374
[2] J. Austin et al., “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732, 2021. https://doi.org/10.48550/arXiv.2108.07732
[3] A. Zia, C. Sharma, V. Giedt, and O. Ahmed, “Generating automated unit tests with LLMs: A practical solution to a key AI challenge,” reportedly IEEE Transactions on Software Engineering, vol. 50, no. 1, pp. 85–105, 2024. [Unverified—do not use without confirming the original publication.]
[4] S. Peng, E. Kalliamvakou, P. Cihon, and M. Demirer, “The impact of AI on developer productivity: Evidence from GitHub Copilot,” arXiv preprint arXiv:2302.06590, 2023. https://doi.org/10.48550/arXiv.2302.06590
[5] A. Fan et al., “Large language models for software engineering: Survey and open problems,” in Proc. 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE), 2023, pp. 31–53. https://doi.org/10.1109/ICSE-FoSE59343.2023.00008
[6] X. Hou et al., “Large language models for software engineering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 8, Art. no. 220, pp. 1–79, 2024. https://doi.org/10.1145/3695988
[7] N. Carlini et al., “Extracting training data from large language models,” in Proc. 30th USENIX Security Symposium (USENIX Security), 2021, pp. 2633–2650.
[8] N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tramèr, and C. Zhang, “Quantifying memorization across neural language models,” in Proc. International Conference on Learning Representations (ICLR), 2023.
[9] H. Pearce, B. Ahmad, B. Tan, B. Dolan-Gavitt, and R. Karri, “Asleep at the keyboard? Assessing the security of GitHub Copilot’s code contributions,” in Proc. 2022 IEEE Symposium on Security and Privacy (SP), 2022, pp. 754–768. https://doi.org/10.1109/SP46214.2022.9833571
[10] J. Liu, C. S. Xia, Y. Wang, and L. Zhang, “Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 36, 2023. https://arxiv.org/abs/2305.01210
[11] A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P. Devanbu, “On the naturalness of software,” in Proc. 34th International Conference on Software Engineering (ICSE), 2012, pp. 837–847. https://doi.org/10.1109/ICSE.2012.6227135
[12] M. Bruch, M. Monperrus, and M. Mezini, “Learning from examples to improve code completion systems,” in Proc. 7th Joint Meeting of the European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE), 2009, pp. 213–222. https://doi.org/10.1145/1595696.1595728
[13] V. Raychev, M. Vechev, and E. Yahav, “Code completion with statistical language models,” in Proc. 35th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI), 2014, pp. 419–428. https://doi.org/10.1145/2594291.2594321
[14] M. Allamanis, E. T. Barr, P. Devanbu, and C. Sutton, “A survey of machine learning for big code and naturalness,” ACM Computing Surveys, vol. 51, no. 4, Art. no. 81, pp. 1–37, 2018. https://doi.org/10.1145/3212695
[15] Y. Bengio, R. Ducharme, P. Vincent, and C. Janvin, “A neural probabilistic language model,” Journal of Machine Learning Research, vol. 3, pp. 1137–1155, 2003.
[16] A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems (NeurIPS), vol. 30, pp. 5998–6008, 2017.
[17] Y. Li et al., “Competition-level code generation with AlphaCode,” Science, vol. 378, no. 6624, pp. 1092–1097, 2022. https://doi.org/10.1126/science.abq1158
[18] A. Holtzman, J. Buys, L. Du, M. Forbes, and Y. Choi, “The curious case of neural text degeneration,” in Proc. International Conference on Learning Representations (ICLR), 2020. https://arxiv.org/abs/1904.09751
[19] B. P. Miller, L. Fredriksen, and B. So, “An empirical study of the reliability of UNIX utilities,” Communications of the ACM, vol. 33, no. 12, pp. 32–44, 1990. https://doi.org/10.1145/96267.96279
[20] V. J. M. Manès et al., “The art, science, and engineering of fuzzing: A survey,” IEEE Transactions on Software Engineering, vol. 47, no. 11, pp. 2312–2331, 2021. https://doi.org/10.1109/TSE.2019.2946563
[21] P. Godefroid, H. Peleg, and R. Singh, “Learn&Fuzz: Machine learning for input fuzzing,” in Proc. 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2017, pp. 50–59. https://doi.org/10.1109/ASE.2017.8115618
[22] C. Lemieux, J. P. Inala, S. K. Lahiri, and S. Sen, “CodaMosa: Escaping coverage plateaus in test generation with pre-trained large language models,” in Proc. 45th IEEE/ACM International Conference on Software Engineering (ICSE), 2023, pp. 919–931. https://doi.org/10.1109/ICSE48619.2023.00085
[23] Z. Yuan et al., “No more manual tests? Evaluating and improving ChatGPT for unit test generation,” arXiv preprint arXiv:2305.04207, 2023. https://doi.org/10.48550/arXiv.2305.04207
[24] Y. Deng, C. S. Xia, H. Peng, C. Yang, and L. Zhang, “Large language models are zero-shot fuzzers: Fuzzing deep-learning libraries via large language models,” in Proc. 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA), 2023, pp. 423–435. https://doi.org/10.1145/3597926.3598067
[25] C. S. Xia, M. Paltenghi, J. L. Tian, M. Pradel, and L. Zhang, “Fuzz4All: Universal fuzzing with large language models,” in Proc. 46th IEEE/ACM International Conference on Software Engineering (ICSE), 2024. https://arxiv.org/abs/2308.04748
[26] A. Ziegler et al., “Productivity assessment of neural code completion,” in Proc. 6th ACM SIGPLAN International Symposium on Machine Programming (MAPS), 2022, pp. 21–29. https://arxiv.org/abs/2205.06537
[27] S. Barke, M. B. James, and N. Polikarpova, “Grounded Copilot: How programmers interact with code-generating models,” Proceedings of the ACM on Programming Languages, vol. 7, no. OOPSLA1, Art. no. 78, pp. 85–111, 2023. https://doi.org/10.1145/3586030
[28] N. Perry, M. Srivastava, D. Kumar, and D. Boneh, “Do users write more insecure code with AI assistants?,” in Proc. 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS), 2023, pp. 2785–2799. https://doi.org/10.1145/3576915.3623157
[29] I. H. Witten and T. Bell, “The zero-frequency problem: Estimating the probabilities of novel events in adaptive text compression,” IEEE Transactions on Information Theory, vol. 37, no. 4, pp. 1085–1094, 1991. https://doi.org/10.1109/18.87000
[30] S. F. Chen and J. Goodman, “An empirical study of smoothing techniques for language modeling,” Computer Speech & Language, vol. 13, no. 4, pp. 359–394, 1999. https://doi.org/10.1006/csla.1999.0128
Downloads
Published
Data Availability Statement
No
Issue
Section
License
Copyright (c) 2026 Suresh Babu Narra (Author)
This work is licensed under a Creative Commons Attribution 4.0 International License.
This journal is licensed under the Creative Commons Attribution 4.0 International License (CC BY 4.0). Authors retain copyright in their published work and permit others to share and adapt the material for any purpose, provided appropriate credit is given to the original author(s) and source, a link to the license is provided, and any changes made are indicated.
