Self-supervision of Hallucinations in Large Language Models: LLteaM

Abstract

Large language models like GPT and Claude have revolutionized the tech industry over the past year. However, as generative artificial intelligence, they are prone to hallucinations. A large language model hallucinates when it generates false or nonsensical text. As these models improve, these hallucinations become less obvious and more dangerous for users. This research explores the phenomenon in the context of automated email response for customer service. First, it proposes a taxonomy of hallucinations in large language models based on their linguistic nature, and second, a multi-agent system that allows for the self-supervision of such hallucinations. This system generates email responses but prevents their delivery if hallucinations are detected, thus reducing the risks of generative AI in productive environments. Experiments with various state-of-the-art language models reveal that the only successful model’s operating costs currently exceed those viable for operational deployment. Moreover, a drastic performance drop after a recent update to GPT-3.5-turbo suggests likely shortcomings in industrial applications driven by retrieval-augmented generation. Overall, the research advocates for a Machine Linguistics to analyze the outputs of large language models, suggesting that such a collaboration between Linguistics and Artificial Intelligence could help mitigate the social risks of hallucination.