Performance Comparison of Sentiment Classification Algorithms on SIGNAL Reviews Using SMOTE
DOI:
https://doi.org/10.51519/journalisi.v7i3.1196Keywords:
sentiment analysis, user reviews, classification algorithm, SMOTE, text classificationAbstract
Public service apps like SIGNAL are widely used to provide public access to information and vehicle tax payments. However, diverse user reviews highlight the need to evaluate public perception through sentiment analysis. Selecting an appropriate classification algorithm is crucial to ensure accurate results, particularly when dealing with imbalanced review data. Therefore, This study examines the comparative performance of four algorithms Naïve Bayes, Random Forest, Decision Tree, and SVM in analyzing the sentiment of 36,000 user feedback obtained from Google Play Store. The dataset underwent preprocessing, feature extraction using TF-IDF, and class balancing using SMOTE. Model evaluation was conducted using accuracy, precision, recall, and F1-score. The findings indicated that Random Forest performed the best overall performance (accuracy 91.04%, F1-score 94.80%), followed by Naïve Bayes (accuracy 89.89%, F1-score 93.38%), SVM (accuracy 89.22%, F1-score 93.02%), and Decision Tree (accuracy 88.40%, F1-score 92.31%). These findings indicate that Random Forest is highly effective for balanced datasets, while SVM and Naïve Bayes offer competitive precision for applications prioritizing accuracy in positive class detection. The output of this study can be applied practically by developers and related institutions in optimizing public service applications and by applying Random Forest algorithm to gain actionable insights for optimizing features and aligning services more closely with user needs.
Downloads
References
C. S. Octiva, P. E. Haes, T. I. Fajri, H. Eldo, and M. L. Hakim, "Implementasi teknologi informasi pada UMKM: Tantangan dan peluang," J. Minfo Polgan, vol. 13, no. 1, pp. 815–821, Jul. 2024.
C. Mauni, “Peran teknologi informasi dalam meningkatkan transparansi dan akuntabilitas pemerintah daerah Jawa Timur,” J. Ilmu Sosial dan Humaniora (ISORA), vol. 3, no. 1, pp. 104–115, Jan.–Jun. 2025.
D. Kurniadi and T. Sutabri, "Penerapan aplikasi Samsat Digital sebagai layanan publik di Sumatera Selatan," Switch: Jurnal Sains dan Teknologi Informasi, vol. 3, no. 1, pp. 49–56, Dec. 2024.
R. Chandra and E. M. Sipayung, “Analisis Sentimen Ulasan Aplikasi Samsat Digital Nasional Menggunakan Algoritma Naive Bayes Classifier,” Jurnal Nasional Teknologi dan Sistem Informasi, vol. 10, no. 3, pp. 156–164, Jan. 2025.
N. Y. Setyawati and E. Widarti, “Analisis sentimen aplikasi Samsat Digital di Play Store menggunakan Support Vector Machine,” J. Smart Syst., vol. 3, no. 2, pp. 33–42, Jan. 2024.
A. S. Kirana, R. Rusdah, R. Roeswidiah, and A. Pudoli, “Analisis sentimen pada media sosial terhadap layanan Samsat Digital Nasional dengan Support Vector Machine,” Idealis: Indonesia Journal Information System, vol. 8, no. 1, pp. 53–63, Jan. 2025.
S. Kacung, C. P. P. Bagyana, and D. Cahyono, “Analisis sentimen terhadap layanan Samsat Digital Nasional (SIGNAL) menggunakan metode SVM,” J. MNEMONIC, vol. 7, no. 1, pp. 118–122, Feb. 2024.
H. M. Puspasari, I. Z. Mustaqim, A. T. Utami, R. Syalevi, and Y. Ruldeviyani, “Evaluation of Indonesia’s police public service platforms through sentiment and thematic analysis,” IAES International Journal of Artificial Intelligence, vol. 13, no. 2, pp. 1596–1607, Jun. 2024.
A. Syukron, E. Saputro, and P. Widodo, “Penerapan Metode Smote Untuk Mengatasi Ketidakseimbangan Kelas Pada Prediksi Gagal Jantung,” J-TIT: j. Teknologi Informasi dan Terapan, Vol. 10 No. 1 Juni 2023
C. Candra, K. W. Chandra, and H. Irsyad, “Efektifitas SMOTE dalam Mengatasi Imbalanced Class Algoritma K-Nearest Neighbors pada Analisis Sentimen terhadap Starlink,” Jurnal Ilmu Komputer dan Informatika, vol. 4, no. 1, pp. 31–42, Jul. 2024.
A. Tripathi, R. Chakraborty, and S. K. Kopparapu, “A Novel Adaptive Minority Oversampling Technique for Improved Classification in Data Imbalanced Scenarios,” arXiv preprint arXiv:2103.13823, 2021.
F. Dwianasari, R. D. Yani, K. N. Laksono, N. Mujaliza, and R. Fahlapi, "Analisis Sentimen Masyarakat terhadap Aktivitas Pertambangan di Raja Ampat Menggunakan Support Vector Machine dan Naïve Bayes dengan Teknik SMOTE," Kajian Ekonomi dan Akuntansi Terapan, vol. 2, no. 2, pp. 234-244, Juni 2025
A. H. Luthfi, A. Faqih, and G. Dwilestari, “Enhancing Model Accuracy in Sentiment Analysis of the by.U Application Using Naïve Bayes and SMOTE Techniques,” J. of Artificial Intelligence and Engineering Applications, vol. 4, no. 2, Feb 2025.
F. Destiyanti, A. I. Hadiana, and F. R. Umbara, "Penerapan Metode Support Vector Machine dan SMOTE untuk Klasifikasi Sentimen Publik Terhadap Polisi Republik Indonesia," JUMANJI, vol. 8, no. 1, pp. 1-15, April 2024.
M. F. Naufal, T. Arifin, and H. Wirjawan, “Analisis perbandingan tingkat performa algoritma SVM, Random Forest, dan Naïve Bayes untuk klasifikasi cyberbullying pada media sosial,” JURASIK: Jurnal Riset Sistem Informasi dan Teknik Informatika, vol. 8, no. 1, pp. 70–81, Feb. 2023.
O. S. D. Fadhillah, J. H. Jaman, and C. Carudin, "Perbandingan Naive Bayes, Support Vector Machine, Logistic Regression dan Random Forest dalam menganalisis sentimen mengenai TikTokshop," J. Inform. dan Tek. Elektro Terapan, vol. 13, no. 1, Jan. 2025.
S. A. H. Bahtiar, C. K. Dewa, and A. Luthfi, “Comparison of Naïve Bayes and Logistic Regression in Sentiment Analysis on Marketplace Reviews Using Rating-Based Labeling,” Journal of Information Systems and Informatics, vol. 5, no. 3, pp. 915–927, Aug. 2023.
S. R. Putri, M. Arifin, and Supriyono, "Analisis Sentimen Publik terhadap Nadiem Makarim sebagai Mendikbudrisktek menggunakan Support Vector Machine (SVM)," Sistemasi: Jurnal Sistem Informasi, vol. 14, no. 2, pp. 826-834, 2025.
T. Dodiya, “Using Term Frequency - Inverse Document Frequency to find the Relevance of Words in Gujarati Language,” Int J Res Appl Sci Eng Technol, vol. 9, no. 4, pp. 378–381, Apr. 2021.
H. Sari, G. L. Ginting, T. Zebua, and Mesran, "Penerapan Algoritma Text Mining dan TF-IDF untuk Pengelompokan Topik Skripsi pada Aplikasi Repository STMIK Budi Darma," TIN: Terapan Informatika Nusantara, vol. 2, no. 7, pp. 414–432, Dec. 2021.
F. Sidik and I. F. Mahdy, "Penerapan SMOTE untuk Mengatasi Imbalance Class dalam Klasifikasi Sentimen Genosida Palestina Menggunakan Multinomial Naive Bayes," Bandung Conference Series Statistics, vol. 4, no. 2, pp. 320-328, Aug. 2024.
E. Espinosa and A. Figueira, "On the quality of synthetic generated tabular data," Mathematics, vol. 11, no. 15, Art. no. 3278, Jul. 2023, doi: 10.3390/math11153278.
R. A. Danquah, "Handling Imbalanced Data: A Case Study for Binary Class Problems," arXiv preprint arXiv:2010.04326, 2020.
A. K. Hidayah, Y. Erwadi, and S. Handayani, "Analisis Sentimen Publik Terhadap Pemindahan Ibu Kota Negara di Twitter Menggunakan Metode Klasifikasi Random Forest dan SMOTE," JATI, vol. 9, no. 5, pp. 9109-9114, Oct. 2025.
I. Muhandhis, A. S. Ritonga, W. Putra, and R. Benowo, “Public sentiment analysis on TikTok about Tapera policy using Random Forest classifier,” Sistemasi: J. Sistem Informasi, vol. 14, no. 1, pp. 354–365, 2025.
Downloads
Published
Issue
Section
License
Authors Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
