NEURAL NETWORK PRUNING IN UNSUPERVISED ASPECT DETECTION BASED ON ASPECT EMBEDDING

• Main Author: Haris Maulana, Muhammad
• Format: Theses
• Language: Indonesia
• Online Access: https://digilib.itb.ac.id/gdl/view/68350
• Institution: Institut Teknologi Bandung

Aspect detection systems for online reviews, mainly based on unsupervised learning, are considered better strategically to process online reviews which can be in a large amount and also unstructured. Deep learning models that utilize aspect embedding, i.e. aspect representation mapping in embedding dimension, have been developed to solve such problems, such as the Attention-based Aspect Extraction Model (He et al. 2017) and Self-Supervised Contrastive Learning Model (Shi et al. 2020). Attention-based Aspect Extraction(ABAE) model with High-Resolution Selective Mapping (Shi et al., 2020) strategy was able to produce a good performance compared to the normal ABAE model. However, those models still use redundant word embedding and are sensitive to initialization changes which may affect their performance. In this study, pruning methodology is used to reduce the model redundant parameters and create new sparse models. This study includes several pruning experiments and comparisons of model performance after pruning. The ABAE model with HRSMap strategy is used as the baseline model to study the effect of pruning of the model. The pruning strategy implemented in the experiments is based on the popular pruning technique (Han et al. 2015) and the lottery ticket hypothesis (Frankle and Carbin 2019). The dataset used for evaluating experiments result is the Citysearch dataset which is used in previous unsupervised aspect categorization model research (He et al. 2017; Shi et al. 2020). The result of this research is that pruning can produce a sparse model of unsupervised aspect embedding-based aspect categorization model with similar performance. However, we have not found a specific pruning method that can consistently improve the performance of the sparse model significantly. A sparse model with only 20% of its original weight can still have comparable performance, with only 1.02% performance difference, with the original model. Our best model from this study is a sparse model created by one-shot pruning their weights, resetting surviving parameters to a randomized model parameters initialization, and fine-tuning the model, all in 2 iterations. Our best model has 80% of total parameters remaining and reaches an F1 score of 0.824, with an increase in performance of 1.01%