In the rush to leverage the latest in AI, many organizations are quickly adopting large language models (LLMs) for a variety of applications, from content generation to complex decision-making processes. The allure of LLMs is undeniable; they offer a seemingly effortless solution to complex problems, and developers can often create such a solution quickly and without needing years of deep machine learning experience. It can be tempting to roll out an LLM-based solution as soon as it’s more or less working and then move on. Although these LLM-based proofs of value are useful, we advise teams to look carefully at what the technology is being used for and to consider whether an LLM is actually the right end-stage solution. Many problems that an LLM can solve — such as sentiment analysis or content classification — can be solved more cheaply and easily using traditional natural language processing (NLP). Analyzing what the LLM is doing and then analyzing other potential solutions not only mitigates the risks associated with overenthusiastic LLM use but also promotes a more nuanced understanding and application of AI technologies.