Integrating LLMs and Academic Writing Instruction: Engaged Reading and LLM-Generated Material

Kamil Lemanek; Aleksander Leczkowski

doi:10.31261/IJREL.2026.12.1.01

Published: 2026-06-30

2026

Integrating LLMs and Academic Writing Instruction: Engaged Reading and LLM-Generated Material

Kamil Lemanek

, Aleksander Leczkowski

Section: Articles

https://doi.org/10.31261/IJREL.2026.12.1.01

Abstract

Integrating Large Language Models (LLMs) into course designs in the higher education context remains an open and quickly growing area of research. This study contributes to the emerging literature by examining how students engage with and evaluate LLM-generated feedback within academic writing instruction. Building on prior work on integration and LLM detection, the study adopts a diachronic design to assess whether students can reliably distinguish between human- and LLM-generated reviews, and how engagement with material may affect perceived classification. Fifty-one undergraduate participants received both human and LLM-generated reviews for essays they authored, completing surveys before and after revising their work. The results indicate that students were initially at random odds in distinguishing human from LLM-generated texts but exhibited statistically significant improvement following engagement. Nearly all students reported positive attitudes toward personalized feedback, irrespective of whether they perceived it to be human or LLM-generated. The findings underscore both the pedagogical potential and evaluative ambiguity of LLM integration in writing instruction.

Keywords:

Higher education , Writing , Detection , Course design

Download files

PDF

Citation rules

Lemanek, K., & Leczkowski, A. (2026). Integrating LLMs and Academic Writing Instruction: Engaged Reading and LLM-Generated Material. International Journal of Research in E-Learning, 1–21. https://doi.org/10.31261/IJREL.2026.12.1.01

Cited by / Share

Licence

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

The Copyright Owners of the submitted texts grant the Reader the right to use the pdf documents under the provisions of the Creative Commons 4.0 International License: Attribution-Share-Alike (CC BY SA). The user can copy and redistribute the material in any medium or format and remix, transform, and build upon the material for any purpose.

1. License

The University of Silesia Press provides immediate open access to journal’s content under the Creative Commons BY-SA 4.0 license (http://creativecommons.org/licenses/by-sa/4.0/). Authors who publish with this journal retain all copyrights and agree to the terms of the above-mentioned CC BY-SA 4.0 license.

2. Author’s Warranties

The author warrants that the article is original, written by stated author/s, has not been published before, contains no unlawful statements, does not infringe the rights of others, is subject to copyright that is vested exclusively in the author and free of any third party rights, and that any necessary written permissions to quote from other sources have been obtained by the author/s.

If the article contains illustrative material (drawings, photos, graphs, maps), the author declares that the said works are of his authorship, they do not infringe the rights of the third party (including personal rights, i.a. the authorization to reproduce physical likeness) and the author holds exclusive proprietary copyrights. The author publishes the above works as part of the article under the licence "Creative Commons Attribution-ShareAlike 4.0 International".

ATTENTION! When the legal situation of the illustrative material has not been determined and the necessary consent has not been granted by the proprietary copyrights holders, the submitted material will not be accepted for editorial process. At the same time the author takes full responsibility for providing false data (this also regards covering the costs incurred by the University of Silesia Press and financial claims of the third party).

3. User Rights

Under the CC BY-SA 4.0 license, the users are free to share (copy, distribute and transmit the contribution) and adapt (remix, transform, and build upon the material) the article for any purpose, provided they attribute the contribution in the manner specified by the author or licensor.

4. Co-Authorship

If the article was prepared jointly with other authors, the signatory of this form warrants that he/she has been authorized by all co-authors to sign this agreement on their behalf, and agrees to inform his/her co-authors of the terms of this agreement.

I hereby declare that in the event of withdrawal of the text from the publishing process or submitting it to another publisher without agreement from the editorial office, I agree to cover all costs incurred by the University of Silesia in connection with my application.

References

Baidoo-Anu, D., & Owusu Ansah, L. (2023). Education in the era of generative artificial intelligence (AI): Understanding the potential benefits of ChatGPT in promoting teaching and learning (SSRN Scholarly Paper No. 4337484). Social Science Research Network. https://doi.org/10.2139/ssrn.4337484
Google Scholar

Bedington, A., Halcomb, E. F., McKee, H. A., Sargent, T., & Smith, A. (2024). Writing with generative AI and human-machine teaming: Insights and recommendations from faculty and students. Computers and Composition, 71, 102833. https://doi.org/10.1016/j.compcom.2024.102833
Google Scholar

Chan, C. K. Y., & Hu, W. (2023). Students’ voices on generative AI: Perceptions, benefits, and challenges in higher education. International Journal of Educational Technology in Higher Education, 20(1), 43. https://doi.org/10.1186/s41239-023-00411-8
Google Scholar

Clark, E., August, T., Serrano, S., Haduong, N., Gururangan, S., & Smith, N. A. (2021). All that’s “human” is not gold: Evaluating human evaluation of generated text. arXiv. https://doi.org/10.48550/arXiv.2107.00061
Google Scholar

Clark, E., & Smith, N. A. (2021). Choose your own adventure: Paired suggestions in collaborative writing for evaluating story generation models. Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 3566–3575. https://doi.org/10.18653/v1/2021.naacl-main.279
Google Scholar

Ding, L., & Zou, D. (2024). Automated writing evaluation systems: A systematic review of Grammarly, Pigai, and Criterion with a perspective on future directions in the age of generative artificial intelligence. Education and Information Technologies, 29(11), 14151–14203. https://doi.org/10.1007/s10639-023-12402-3
Google Scholar

Du, H., Jia, Q., Gehringer, E., & Wang, X. (2024). Harnessing large language models to auto-evaluate the student project reports. Computers and Education: Artificial Intelligence, 7, 100268. https://doi.org/10.1016/j.caeai.2024.100268
Google Scholar

Dugan, L., Ippolito, D., Kirubarajan, A., Shi, S., & Callison-Burch, C. (2023). Real or fake text? Investigating human ability to detect boundaries between human-written and machine-generated text. Proceedings of the AAAI Conference on Artificial Intelligence, 37(11), 12763–12771. https://doi.org/10.1609/aaai.v37i11.26501
Google Scholar

Gabriel, I., Manzini, A., Keeling, G., Hendricks, L. A., Rieser, V., Iqbal, H., Tomašev, N., Ktena, I., Kenton, Z., Rodriguez, M., El-Sayed, S., Brown, S., Akbulut, C., Trask, A., Hughes, E., Bergman, A. S., Shelby, R., Marchal, N., Griffin, C., … Manyika, J. (2024). The ethics of advanced AI assistants. arXiv. https://doi.org/10.48550/ARXIV.2404.16244
Google Scholar

Grassini, S. (2023). Shaping the future of education: Exploring the potential and consequences of AI and ChatGPT in educational settings. Education Sciences, 13(7), 692. https://doi.org/10.3390/educsci13070692
Google Scholar

Howcroft, D. M., Belz, A., Clinciu, M.-A., Gkatzia, D., Hasan, S. A., Mahamood, S., Mille, S., Van Miltenburg, E., Santhanam, S., & Rieser, V. (2020). Twenty years of confusion in human evaluation: NLG needs evaluation sheets and standardised definitions. Proceedings of the 13th International Conference on Natural Language Generation, 169–182. https://doi.org/10.18653/v1/2020.inlg-1.23
Google Scholar

Jeon, J., & Lee, S. (2023). Large language models in education: A focus on the complementary relationship between human teachers and ChatGPT. Education and Information Technologies, 28(12), 15873–15892. https://doi.org/10.1007/s10639-023-11834-1
Google Scholar

Köbis, N., & Mossink, L. D. (2021). Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry. Computers in Human Behavior, 114, 106553. https://doi.org/10.1016/j.chb.2020.106553
Google Scholar

Liu, W. (2024). A systematic review of automated writing evaluation feedback: Validity, effects and students’ engagement. Language Teaching Research Quarterly, 45, 86–105. https://doi.org/10.32038/ltrq.2024.45.05
Google Scholar

Ma, Y., Liu, J., Yi, F., Cheng, Q., Huang, Y., Lu, W., & Liu, X. (2023). AI vs. human—Differentiation analysis of scientific content generation (Version 2). arXiv. https://doi.org/10.48550/ARXIV.2301.10416
Google Scholar

Novikova, J., Dušek, O., & Rieser, V. (2018). RankME: Reliable human ratings for natural language generation. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), 72–78. https://doi.org/10.18653/v1/n18-2012
Google Scholar

Ortiz-Bonnin, S., & Blahopoulou, J. (2025). Chat or cheat? Academic dishonesty, risk perceptions, and ChatGPT usage in higher education students. Social Psychology of Education, 28(1), 113. https://doi.org/10.1007/s11218-025-10080-2
Google Scholar

Phokoye, S. P., Dlamini, S., Mthalane, P. P., Luthuli, M., & Moyane, S. P. (2025). A comprehensive review of ChatGPT in teaching and learning within higher education. Informatics, 12(3), 74. https://doi.org/10.3390/informatics12030074
Google Scholar

Prunkl, C. E. A., Ashurst, C., Anderljung, M., Webb, H., Leike, J., & Dafoe, A. (2021). Institutionalizing ethics in AI through broader impact requirements. Nature Machine Intelligence, 3(2), 104–110. https://doi.org/10.1038/s42256-021-00298-y
Google Scholar

Qian, Y. (2025). Pedagogical applications of generative AI in higher education: A systematic review of the field. TechTrends. https://doi.org/10.1007/s11528-025-01100-1
Google Scholar

Rosas, R. A. P., & Rodríguez, E. J. A. (2024). Chatbots: The future of education? International Journal of Engineering Pedagogy (iJEP), 14(8), 107–119. https://doi.org/10.3991/ijep.v14i8.49715
Google Scholar

Soni, M., & Wade, V. (2023). Comparing abstractive summaries generated by ChatGPT to real summaries through blinded reviewers and text classification algorithms. arXiv. https://doi.org/10.48550/arXiv.2303.17650
Google Scholar

Stöhr, C., Ou, A. W., & Malmström, H. (2024). Perceptions and usage of AI chatbots among students in higher education across genders, academic levels and fields of study. Computers and Education: Artificial Intelligence, 7, 100259. https://doi.org/10.1016/j.caeai.2024.100259
Google Scholar

Su, Y., & Wu, Y. (2024). Robust detection of LLM-generated text: A comparative analysis. arXiv. https://doi.org/10.48550/arXiv.2411.06248
Google Scholar

Tzirides, A. O. (Olnancy), Zapata, G., Kastania, N. P., Saini, A. K., Castro, V., Ismael, S. A., You, Y., Santos, T. A. D., Searsmith, D., O’Brien, C., Cope, B., & Kalantzis, M. (2024). Combining human and artificial intelligence for enhanced AI literacy in higher education. Computers and Education Open, 6, 100184. https://doi.org/10.1016/j.caeo.2024.100184
Google Scholar

van der Lee, C., Gatt, A., van Miltenburg, E., & Krahmer, E. (2021). Human evaluation of automatically generated text: Current trends and best practice guidelines. Computer Speech & Language, 67, 101151. https://doi.org/10.1016/j.csl.2020.101151
Google Scholar

Wang, H., Li, J., & Li, Z. (2024). AI-Generated text detection and classification based on BERT deep learning algorithm. https://doi.org/10.48550/ARXIV.2405.16422
Google Scholar

Weber-Wulff, D., Anohina-Naumeca, A., Bjelobaba, S., Foltýnek, T., Guerrero-Dib, J., Popoola, O., Šigut, P., & Waddington, L. (2023). Testing of detection tools for AI-generated text. International Journal for Educational Integrity, 19(1), 26. https://doi.org/10.1007/s40979-023-00146-z
Google Scholar

Wu, J., Yang, S., Zhan, R., Yuan, Y., Chao, L. S., & Wong, D. F. (2025). A survey on LLM-generated text detection: Necessity, methods, and future directions. Computational Linguistics, 51(1), 275–338. https://doi.org/10.1162/coli_a_00549
Google Scholar

Wu, J., Zhan, R., Wong, D. F., Yang, S., Yang, X., Yuan, Y., & Chao, L. S. (2024). DetectRL: Benchmarking LLM-generated text detection in real-world scenarios. Advances in Neural Information Processing Systems, 37, 100369–100401.
Google Scholar

Yang, H., Kim, H., Lee, J. H., & Shin, D. (2022). Implementation of an AI chatbot as an English conversation partner in EFL speaking classes. ReCALL, 34(3), 327–343. https://doi.org/10.1017/S0958344022000039
Google Scholar

2026
Published: 2020-07-09

ISSN: 2451-2583

eISSN: 2543-6155

Publisher

Wydawnictwo Uniwersytetu Śląskiego | University of Silesia Press

Licence CC

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

Submit