As late 2022, artificial intelligence (AI) and LLMs have become increasingly prevalent in software engineering. Though, some experienced engineers are observing troubling trends regarding thier use.

While LLMs are often viewed as beneficial tools that enhance productivity, this outlook overlooks significant risks. The allure of rapid code generation can overshadow the importance of careful consideration and thorough understanding.

Potential Pitfalls of LLMs in Software Progress

Over-reliance on LLMs introduces several potential risks:

• Incorrect Output: LLMs can produce code that is either blatantly incorrect (e.g., failing to compile) or, more insidiously, subtly flawed with logic bugs. This risk is amplified when individuals lacking sufficient expertise, such as project managers, use LLMs to generate code.
• Flawed Input: LLMs do not challenge prompts that are leading or based on flawed assumptions or incomplete context. For example, an engineer might ask for a thread-safe list implementation in C#, receiving a technically correct but ultimately unhelpful solution. The LLM doesn’t recognize the underlying problem or suggest a simpler, more appropriate solution using existing libraries.
• Codebase Degradation: Overuse of AI can rapidly degrade the quality of a codebase, leading to a situation akin to hoarding disorder where the internal structure becomes unsanitary and dysfunctional.
• Skill Atrophy: Outsourcing problem-solving to LLMs can lead to a decline in critical thinking skills among both senior and junior engineers. Senior engineers lose opportunities for learning through productive struggle, while junior engineers never develop these skills in the first place. “Microsoft research on knowledge workers found that AI-driven confidence often comes at the expense of critical thinking,” according to a recent study.
• Diminished Job Satisfaction: Many developers report that using AI diminishes their sense of “flow state” and the joy of creation. AI-generated code can be difficult to read and modify, leading to frustration.

“LLMs give me finished thoughts, polished and convincing, but none of the intellectual growth that comes from developing them myself”

While mitigations for these risks exist, two programming competencies remain uniquely human: program theory and program entropy management.

Program Theory: The Human Element

Peter Naur, a pioneer in computer science, argued that a program is not merely its source code but a shared mental construct – a “theory” or “design.” The code is derived from this design, making the design the true work product of value. As Naur