Document clustering based on inverse document frequency measure

• Main Author: Wan Faridah Hanum, Wan Yaacob
• Format: Thesis
• Language: English; English
• Published: 2005
• Subjects: HF5001-6182 Business
• Online Access: http://etd.uum.edu.my/1367/1/WAN_FARIDAH_HANUM_BT._WAN_YAACOB.pdf; http://etd.uum.edu.my/1367/2/1.WAN_FARIDAH_HANUM_BT._WAN_YAACOB.pdf; http://etd.uum.edu.my/1367/; http://sierra.uum.edu.my/record=b1170635~S1
• Institution: Universiti Utara Malaysia

Automatic classification techniques are capable of providing the necessary information organization by arranging the retrieved data into groups of documents with common subjects. Recently, document clustering has been put forth as an alternative method of organizing the results of retrieval. It been proposed for use in navigating and browsing document collections, and discovers hidden similarity and key concepts. It also summarize a large amount of document using key or common attributes of cluster and can be used to categorize document databases. This paper describes several narrative clustering techniques such as Porter algorithm, Gusfield algorithm, similarity based on document hierarchy and Inverse Document Frequency (IDF), which intersect the documents in a cluster to determine the set of words (or phrases) shared by all the documents in the cluster. This study proposes document clustering based on IDF, where it is assumes that importance of a keyword in calculating similarity measures is inversely proportional to the total number of documents that contain it. IDF is easy to understand, has a geometric interpretation, term weighing shown to help clustering, allow partial matching and returns ranked documents. An important finding in this study, where 30 cases of documents tested with the IDF algorithm, and the results are divided into three category; correct cluster, incorrect cluster, and unknown cluster.