A Novel Approach: Tokenization Framework based on Sentence Structure in Indonesian Language

Sukemi, Sukemi and petrus, johannes and ermatita, ermatita (2023) A Novel Approach: Tokenization Framework based on Sentence Structure in Indonesian Language. International Journal of Advanced Computer Science and Applications(IJACSA), 14 (2). pp. 541-549. ISSN 2158-107X (Print)

Text (Jurnal Internasional terindeks SCOPUS) Paper-IJACSA.pdf - Published Version Download (753kB)

Abstract

This study proposes a new approach in the sentence tokenization process. Sentence tokenization, which is known so far, is the process of breaking sentences based on spaces as separators. Space-based sentence tokenization only generates single word tokens. In sentences consisting of five words, tokenization will produce five tokens, one word each. Each word is a token. This process ignores the loss of the original meaning of the separated words. Our proposed tokenization framework can generate one-word tokens and multi-word tokens at the same time. The process is carried out by extracting the sentence structure to obtain sentence elements. Each sentence element is a token. There are five sentence elements that is Subject, Predicate, Object, Complement and Adverbs. We extract sentence structures using deep learning methods, where models are built by training the datasets that have been prepared before. The training results are quite good with an F1 score of 0.7 and it is still possible to improve. Sentence similarity is the topic for measuring the performance of one-word tokens compared to multi-word tokens. In this case the multiword token has better accuracy. This framework was created using the Indonesian language but can also use other languages with dataset adjustments.

Item Type:	Article
Subjects:	Q Science > Q Science (General) > Q334-342 Computer science. Artificial intelligence. Algorithms. Robotics. Automation.
Divisions:	09-Faculty of Computer Science > 56201-Computer Systems (S1)
Depositing User:	Dr. Sukemi Sukemi
Date Deposited:	11 Apr 2023 13:56
Last Modified:	17 Apr 2023 02:17
URI:	http://repository.unsri.ac.id/id/eprint/95818

Actions (login required)

View Item