Abstract

Educators have been concerned about the capability of large language models to automatically generate programs in response to textual prompts. However, little is known about whether and how students actually use these tools.
In the context of an upper-level formal methods course, we gave students access to large language models. They were told they could use the models freely. We built a Visual Studio Code extension to simplify access to these models. We also paid for an account so students could use the models for free without worrying about cost.
In this experience report we analyze the outcomes. We see how students actually do and do not use the models. We codify the different uses they make. Most of all, we notice that students actually do not use them very much at all, and provide insight into the many reasons why not. We believe such experiments can help rebalance some of the public narrative about such tools.