An evaluation of tokenizers on domain specific text
The healthcare industry is fast realizing the value of data, collecting information from electronic health record systems (EHRs), sensors, and other sources. However, the problem of understanding the data collected in the process has been existed for years. According to big data analytics in heal...
Saved in:
Main Author: | |
---|---|
Other Authors: | |
Format: | Final Year Project |
Language: | English |
Published: |
Nanyang Technological University
2022
|
Subjects: | |
Online Access: | https://hdl.handle.net/10356/156461 |
Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Institution: | Nanyang Technological University |
Language: | English |
Summary: | The healthcare industry is fast realizing the value of data, collecting information from
electronic health record systems (EHRs), sensors, and other sources. However, the
problem of understanding the data collected in the process has been existed for years.
According to big data analytics in healthcare, up to 80% of healthcare documentation
is unstructured and hence generally unutilized, because mining and extracting this data
is challenging and resource intensive.
This is where Natural Language Processing can come in. NLP technology services
have the potential to extract meaningful insights and concepts from data that was
previously considered buried in text form.
In NLP studies, text preprocessing is traditionally the first step in building a Machine
Learning model, and in the process of text preprocessing, the very first and usually the
most important step is tokenization. Currently, many open-source tools for
tokenization are available for tokenizing text based on different rules, but few studies
have been done on the performance of tokenizers on domain specific text—e.g.,
healthcare domain.
Therefore, this project aims to, first, evaluate different open-source tokenizers’
performance on medical text data and select the best-performing tokenizer; after that,
build a wrapper based on the best-performing tokenizer, to further improve its
performance on medical text data.
In this way, more accurate tokenization results of medical text data can be achieved,
and these results can be used in the following NLP process to generate more
meaningful insights. With NLP technology, physicians can enhance patient care,
research efforts, and disease diagnosis methods. |
---|