Toolkit development for high-dimensional data pre-processing, clustering and analysis

• Main Author: Hu, Yao.
• Other Authors: Chen Lihui
• Format: Theses and Dissertations
• Language: English
• Published: 2014
• Subjects: DRNTU::Engineering::Electrical and electronic engineering
• Online Access: http://hdl.handle.net/10356/55243
• Institution: Nanyang Technological University

In this report, the author documents the software project that designs and implements a high dimensional data processing toolkit. The developed toolkit is called WordTagger, that automatically labels a vocabulary of computer science words to provide the categorical information of the word-space by using ACM taxonomy as reference [1]. The word categorical information can be used as another source of the prior knowledge to incorporate with that from the document-space into the existing semi-supervised coclustering algorithms. The author has successfully implemented this toolkit WordTagger and conducted tests to evaluate its effectiveness and efficiency. Some preliminary experiments have also been conducted to show the WordTagger labeled words could be used as an additional word-space prior knowledge source. This is done by making modifications to an existing semi-supervised approach SS-HFCR to accept prior knowledge from both document and word-space, which is referred as dual SS-HFCR. However, in the report, we show that dual SS-HFCR is unable to perform as good as expected with the categorical information from word-space provided by WordTagger. The limitations of the current integration of WordTagger and dual SS-HFCR are identified and discussed. The future work is suggested and summarized in the end of the report.