Performance Comparison of Sentiment Classification Algorithms on SIGNAL Reviews Using SMOTE

Qothrunnada Wafi Anadia; Allsela Meiriza

doi:10.51519/journalisi.v7i3.1196

Qothrunnada Wafi Anadia Sriwijaya University, Indonesia
Allsela Meiriza Sriwijaya University, Indonesia

DOI: 10.51519/journalisi.v7i3.1196

Keywords: sentiment analysis, user reviews, classification algorithm, SMOTE, text classification

Abstract

Public service apps like SIGNAL are widely used to provide public access to information and vehicle tax payments. However, diverse user reviews highlight the need to evaluate public perception through sentiment analysis. Selecting an appropriate classification algorithm is crucial to ensure accurate results, particularly when dealing with imbalanced review data. Therefore, This study examines the comparative performance of four algorithms Naïve Bayes, Random Forest, Decision Tree, and SVM in analyzing the sentiment of 36,000 user feedback obtained from Google Play Store. The dataset underwent preprocessing, feature extraction using TF-IDF, and class balancing using SMOTE. Model evaluation was conducted using accuracy, precision, recall, and F1-score. The findings indicated that Random Forest performed the best overall performance (accuracy 91.04%, F1-score 94.80%), followed by Naïve Bayes (accuracy 89.89%, F1-score 93.38%), SVM (accuracy 89.22%, F1-score 93.02%), and Decision Tree (accuracy 88.40%, F1-score 92.31%). These findings indicate that Random Forest is highly effective for balanced datasets, while SVM and Naïve Bayes offer competitive precision for applications prioritizing accuracy in positive class detection. The output of this study can be applied practically by developers and related institutions in optimizing public service applications and by applying Random Forest algorithm to gain actionable insights for optimizing features and aligning services more closely with user needs.

Downloads

Download data is not yet available.

References

C. S. Octiva, P. E. Haes, T. I. Fajri, H. Eldo, and M. L. Hakim, "Implementasi teknologi informasi pada UMKM: Tantangan dan peluang," J. Minfo Polgan, vol. 13, no. 1, pp. 815–821, Jul. 2024.

C. Mauni, “Peran teknologi informasi dalam meningkatkan transparansi dan akuntabilitas pemerintah daerah Jawa Timur,” J. Ilmu Sosial dan Humaniora (ISORA), vol. 3, no. 1, pp. 104–115, Jan.–Jun. 2025.

D. Kurniadi and T. Sutabri, "Penerapan aplikasi Samsat Digital sebagai layanan publik di Sumatera Selatan," Switch: Jurnal Sains dan Teknologi Informasi, vol. 3, no. 1, pp. 49–56, Dec. 2024.

R. Chandra and E. M. Sipayung, “Analisis Sentimen Ulasan Aplikasi Samsat Digital Nasional Menggunakan Algoritma Naive Bayes Classifier,” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 10, no. 3, pp. 156–164, Jan. 2025.

N. Y. Setyawati and E. Widarti, “Analisis sentimen aplikasi Samsat Digital di Play Store menggunakan Support Vector Machine,” J. Smart Syst., vol. 3, no. 2, pp. 33–42, Jan. 2024.

A. S. Kirana, R. Rusdah, R. Roeswidiah, and A. Pudoli, “Analisis sentimen pada media sosial terhadap layanan Samsat Digital Nasional dengan Support Vector Machine,” Idealis: Indonesia Journal Information System, vol. 8, no. 1, pp. 53–63, Jan. 2025.

S. Kacung, C. P. P. Bagyana, and D. Cahyono, “Analisis sentimen terhadap layanan Samsat Digital Nasional (SIGNAL) menggunakan metode SVM,” J. MNEMONIC, vol. 7, no. 1, pp. 118–122, Feb. 2024.

H. M. Puspasari, I. Z. Mustaqim, A. T. Utami, R. Syalevi, and Y. Ruldeviyani, “Evaluation of Indonesia’s police public service platforms through sentiment and thematic analysis,” IAES International Journal of Artificial Intelligence, vol. 13, no. 2, pp. 1596–1607, Jun. 2024.

A. Syukron, E. Saputro, and P. Widodo, “Penerapan Metode Smote Untuk Mengatasi Ketidakseimbangan Kelas Pada Prediksi Gagal Jantung,” J-TIT: j. Teknologi Informasi dan Terapan, Vol. 10 No. 1 Juni 2023

C. Candra, K. W. Chandra, and H. Irsyad, “Efektifitas SMOTE dalam Mengatasi Imbalanced Class Algoritma K-Nearest Neighbors pada Analisis Sentimen terhadap Starlink,” Jurnal Ilmu Komputer dan Informatika, vol. 4, no. 1, pp. 31–42, Jul. 2024.

A. Tripathi, R. Chakraborty, and S. K. Kopparapu, “A Novel Adaptive Minority Oversampling Technique for Improved Classification in Data Imbalanced Scenarios,” arXiv preprint arXiv:2103.13823, 2021.

F. Dwianasari, R. D. Yani, K. N. Laksono, N. Mujaliza, and R. Fahlapi, "Analisis Sentimen Masyarakat terhadap Aktivitas Pertambangan di Raja Ampat Menggunakan Support Vector Machine dan Naïve Bayes dengan Teknik SMOTE," Kajian Ekonomi dan Akuntansi Terapan, vol. 2, no. 2, pp. 234-244, Juni 2025

A. H. Luthfi, A. Faqih, and G. Dwilestari, “Enhancing Model Accuracy in Sentiment Analysis of the by.U Application Using Naïve Bayes and SMOTE Techniques,” J. of Artificial Intelligence and Engineering Applications, vol. 4, no. 2, Feb 2025.

F. Destiyanti, A. I. Hadiana, and F. R. Umbara, "Penerapan Metode Support Vector Machine dan SMOTE untuk Klasifikasi Sentimen Publik Terhadap Polisi Republik Indonesia," JUMANJI, vol. 8, no. 1, pp. 1-15, April 2024.

M. F. Naufal, T. Arifin, and H. Wirjawan, “Analisis perbandingan tingkat performa algoritma SVM, Random Forest, dan Naïve Bayes untuk klasifikasi cyberbullying pada media sosial,” JURASIK: Jurnal Riset Sistem Informasi dan Teknik Informatika, vol. 8, no. 1, pp. 70–81, Feb. 2023.

O. S. D. Fadhillah, J. H. Jaman, and C. Carudin, "Perbandingan Naive Bayes, Support Vector Machine, Logistic Regression dan Random Forest dalam menganalisis sentimen mengenai TikTokshop," J. Inform. dan Tek. Elektro Terapan, vol. 13, no. 1, Jan. 2025.

S. A. H. Bahtiar, C. K. Dewa, and A. Luthfi, “Comparison of Naïve Bayes and Logistic Regression in Sentiment Analysis on Marketplace Reviews Using Rating-Based Labeling,” Journal of Information Systems and Informatics, vol. 5, no. 3, pp. 915–927, Aug. 2023.

S. R. Putri, M. Arifin, and Supriyono, "Analisis Sentimen Publik terhadap Nadiem Makarim sebagai Mendikbudrisktek menggunakan Support Vector Machine (SVM)," Sistemasi: Jurnal Sistem Informasi, vol. 14, no. 2, pp. 826-834, 2025.

T. Dodiya, “Using Term Frequency - Inverse Document Frequency to find the Relevance of Words in Gujarati Language,” Int J Res Appl Sci Eng Technol, vol. 9, no. 4, pp. 378–381, Apr. 2021.

H. Sari, G. L. Ginting, T. Zebua, and Mesran, "Penerapan Algoritma Text Mining dan TF-IDF untuk Pengelompokan Topik Skripsi pada Aplikasi Repository STMIK Budi Darma," TIN: Terapan Informatika Nusantara, vol. 2, no. 7, pp. 414–432, Dec. 2021.

F. Sidik and I. F. Mahdy, "Penerapan SMOTE untuk Mengatasi Imbalance Class dalam Klasifikasi Sentimen Genosida Palestina Menggunakan Multinomial Naive Bayes," Bandung Conference Series Statistics, vol. 4, no. 2, pp. 320-328, Aug. 2024.

E. Espinosa and A. Figueira, "On the quality of synthetic generated tabular data," Mathematics, vol. 11, no. 15, Art. no. 3278, Jul. 2023, doi: 10.3390/math11153278.

R. A. Danquah, "Handling Imbalanced Data: A Case Study for Binary Class Problems," arXiv preprint arXiv:2010.04326, 2020.

A. K. Hidayah, Y. Erwadi, and S. Handayani, "Analisis Sentimen Publik Terhadap Pemindahan Ibu Kota Negara di Twitter Menggunakan Metode Klasifikasi Random Forest dan SMOTE," JATI, vol. 9, no. 5, pp. 9109-9114, Oct. 2025.

I. Muhandhis, A. S. Ritonga, W. Putra, and R. Benowo, “Public sentiment analysis on TikTok about Tapera policy using Random Forest classifier,” Sistemasi: J. Sistem Informasi, vol. 14, no. 1, pp. 354–365, 2025.