SARCASM DETECTION IN ENGLISH TEXT USING LLAMA 2 MODEL

• Main Author: Dennis Heraldi, Fachry
• Format: Final Project
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/82487
• Institution: Institut Teknologi Bandung

Detecting sarcasm in English text is a challenge in sentiment analysis systems because sarcastic text implies a meaning different from what is explicitly conveyed. This research aims to build models for sarcasm detection, sarcasm category classification, and pairwise sarcasm identification using fine-tuned Llama 2 models. The methods used include data collection from the main iSarcasmEval dataset and external datasets, with a total combined data of 21,599 for sarcasm detection, 3,457 for sarcasm category classification, and 868 for pairwise sarcasm identification. Furthermore, prompt development, model fine-tuning using Parameter Efficient Fine-tuning (PEFT) technique with the specific method of Quantized Low Rank Adaptation (QLoRA), and model testing with a zero-shot approach were conducted. Lastly, analysis of experimental results and quantitative and qualitative evaluations were performed to draw research conclusions. Experimental results show that the fine-tuned Llama 2 13B model with the combined dataset achieved a sarcastic F1-score of 0.6867 on the sarcasm detection subtask, surpassing the performance of the best team model in iSarcasmEval. On the sarcasm category classification subtask, the fine-tuned Llama 2 7B model achieved a Macro-F1 of 0.1388, better than the performance of the second-best team model in iSarcasmEval. On the pairwise sarcasm identification subtask, the Llama 2 13B model achieved an accuracy of 0.9, surpassing the performance of the best team model in iSarcasmEval. This research proves that the Llama 2 model can improve sarcasm detection performance with the addition of external datasets and the use of appropriate prompt engineering techniques. This research also shows that the PEFT technique with the QLoRA method can reduce memory requirements without sacrificing performance, enabling model development on devices with limited computational resources. The difference in perception in labeling sarcasm datasets remains a major challenge in sarcasm detection, highlighting the importance of context and intention in sarcasm detection.