MITIGATING HALLUCINATION IN AUTOMATED INTERVIEWS USING A CHAIN-OF-VERIFICATION APPROACH ON LLMS

• Main Author: Bintang Nurmansyah, Ilham
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/85547
• Institution: Institut Teknologi Bandung

The use of Large Language Models (LLMs) in recruitment processes presents significant challenges, primarily due to the inherent tendency of LLMs to produce hallucinations. This research develops an anti-hallucination component based on the chain-of-verification method, utilizing GPT-3.5 and GPT-4 models to detect and prevent LLM interviewers from delivering hallucinated responses to candidates. GPT models were selected due to their lower hallucination rates compared to other models at the time of this study (GPT-3.5 at 3.5% and GPT-4 at 3%). The study employs two GPT models to compare the effectiveness of hallucination prevention. The system implementation uses the chain-of-verification technique, applying few-shot learning to the LLM verifier model to identify and differentiate potential hallucinations. This method is integrated as a module within the LLM interviewer, which is invoked each time the LLM interviewer generates a question or response to a candidate's prompt. The results demonstrate that interviews without the chain-of-verification method exhibit an average hallucination rate of 100% when given certain prompts, whereas, with the chain- of-verification, the average hallucination rate is reduced to 4% under the same conditions. GPT-4 emerges as the most effective model, with an average hallucination rate of 12%, compared to 72% for GPT-3.5.