Disentangling syntactics, semantics, and pragmatics in natural language processing

• Main Author: Zhang, Xulang
• Other Authors: Erik Cambria
• Format: Thesis-Doctor of Philosophy
• Language: English
• Published: Nanyang Technological University 2024
• Subjects: Computer and Information Science
• Online Access: https://hdl.handle.net/10356/177426
• Institution: Nanyang Technological University

In the era of deep learning, the natural language processing (NLP) community has become increasingly reliant on large language models (LLM), which are essentially probabilistic black-boxes. Hence, when applying them to downstream applications, we can only entrust that the LLMs has learned the various linguistic phenomena in human language implicitly through extensive pretraining. Although LLMs are versatile and competent for many NLP tasks, they have yet to achieve human-like natural language understanding due to their stochastic nature. Additionally, LLMs offer limited interpretability and trustworthiness, making it difficult for humans to be involved in their learning process. To advance NLP studies, it is worth investigating whether a human-centered, neurosymbolic approach can address these issues. One way to combat the limitations of LLMs is task decomposition, i.e., deconstructing a downstream task into its constituent subtasks. Such approach allows the symbolic representations of different linguistic layers, namely syntactics, semantics, and pragmatics, to be incorporated into the subsymbolic neural model. Specifically, syntactic layer extracts meanings from sentence constituents, and establish sentence structure.Semantic layer deconstructs texts into concepts, and resolves references. Pragmatic layer is then able to extract meaning from both sentence structure and semantics obtained from prior layers.As such, in this thesis, we disentangle the syntactics, semantics, and pragmatics in NLP by studying tasks within each linguistic layer. First, we investigate how syntactic tasks of different granularity might complement each other in a multitask learning setting. The tasks of study include sentence boundary disambiguation, text chunking, and part-of-speech tagging. We propose a soft-parameter-sharing multitask learning model that share local and global dependency information between tasks, and a curriculum learning mechanism to address the non-parallel label data problem. We evaluate our methods on the benchmark datasets, and provide findings on the pair-wise complementarity of the involved tasks. Second, we examine the effective application of semantic processing in downstream task, i.e., sentiment analysis. Word sense disambiguation is a well-studied semantic task, whose application to downstream tasks has been limited by nature of its task setup. By reframing the disambiguation of word sense as a lexical substitution problem that replaces ambiguous words with more definitive ones, we propose a neurosymbolic sentiment analysis method that enables an explicit, dynamic, and explainable incorporation of word sense into the sentiment analysis task. We conduct experiments to prove that our framework is indeed effective in improving the performance of sentiment analysis. Third, using sentiment analysis as a case study, we propose a sentiment knowledge base as a source of pragmatic symbolic representations. Our knowledge base differs from existing sentiment-annotated knowledge bases because it is human-centric and domain-adaptable. It accommodates varying sentiment perceptions of a word in different domains through the use of multi-dimensional representation, addressing the neglect of human subjectivity and disagreement in the prevalent single-score annotation paradigm. Lastly, we present a shallow neurosymbolic model that alleviates the heaviness and rigidness of deep neural network and LLMs by coupling pragmatic and syntactic representations. The former is the sentiment-pragmatic representations from our proposed sentiment knowledge base, and the latter is the part-of-speech sequence of the input, provided by off-the-shelf tagger. Our model stands out for its lightweight nature, robustness across domains and languages, efficient few-shot training, and rapid convergence. We conduct extensive experiments to both show the effectiveness and robustness of our lightweight neurosymbolic framework, as well as the validity of our SenticVec knowledge base. In summary, this thesis deconstructed NLP into syntactic, semantic, and pragmatic processing, and proposed novel solutions for tasks in each categories with an outlook towards human-centric neurosymbolic NLP.