Reliability-Aware Public Issue Priority Mapping for Indonesia’s Free Nutritious Meal Program Using Sentiment Calibration and BERTopic
DOI:
https://doi.org/10.63158/journalisi.v8i4.1753Keywords:
Probability Calibration, BERTopic, Public Issue Priority Mapping, Policy Monitoring, Free Nutritious Meal ProgramAbstract
The Free Nutritious Meal Program (MBG) has generated extensive public discussion on platform X. This study develops a reliability-aware framework for mapping public issue priorities by integrating sentiment analysis, probability calibration, and topic modeling. A final dataset of 1,641 public X posts was de-identified, preprocessed, relevance-filtered, manually labeled, validated, and proportionally split. TF-IDF + SVM was used as a classical baseline, while IndoBERTweet was fine-tuned and calibrated using temperature scaling. BERTopic was applied to generate topics, followed by manual interpretation into substantive issue groups. TF-IDF + SVM outperformed IndoBERTweet, achieving 0.8138 accuracy and 0.7999 Macro-F1, while IndoBERTweet achieved 0.7611 accuracy and 0.7142 Macro-F1. IndoBERTweet was retained because it provides probability-based confidence scores for calibration and priority mapping. Calibration modestly reduced NLL from 0.5239 to 0.5199 and ECE from 0.0700 to 0.0682. BERTopic produced 19 non-noise topics with a coherence score of 0.4552. The highest-priority public discussion theme concerned food safety, poisoning-related discourse, and consumption quality. This framework provides initial public-opinion monitoring input, not definitive policy evaluation, and requires external validation before formal policy use.
Downloads
References
[1] Kementerian Keuangan Republik Indonesia, “Anggaran Makan Bergizi Gratis Rp71 triliun masuk postur RAPBN 2025.” Accessed: Apr. 26, 2026.
[2] M. Lopreite, M. Misuraca, and M. Puliga, “Outbreak and integration of social media in public health surveillance systems: A policy review through BERT embedding technique,” Socioecon. Plann. Sci., vol. 95, Art. no. 101995, Oct. 2024, doi: 10.1016/j.seps.2024.101995.
[3] S. Verma, “Sentiment analysis of public services for smart society: Literature review and future research directions,” Gov. Inf. Q., vol. 39, no. 3, Art. no. 101708, Jul. 2022, doi: 10.1016/j.giq.2022.101708.
[4] A. Huwaidah, Adiwijaya, and S. Al Faraby, “Argument identification in Indonesian tweets on the issue of moving the Indonesian capital,” Procedia Comput. Sci., vol. 179, pp. 407–415, 2021, doi: 10.1016/j.procs.2021.01.023.
[5] A. Mohan, A. M. Nair, B. Jayakumar, and S. Muraleedharan, “Sarcasm detection using Bidirectional Encoder Representations from Transformers and Graph Convolutional Networks,” Procedia Comput. Sci., vol. 218, pp. 93–102, 2023, doi: 10.1016/j.procs.2022.12.405.
[6] R. Saputra, Y. Pristyanto, and I. N. Fajri, “Generative AI image sentiment analysis on social media X using TF-IDF and FastText,” J. Appl. Inform. Comput., vol. 9, no. 5, pp. 2509–2520, Oct. 2025, doi: 10.30871/jaic.v9i5.10627.
[7] E. Supriyadi and P. N. Makatita, “Sentiment analysis of TikTok user comments on QRIS adoption in Indonesia using IndoBERT,” Procedia Comput. Sci., vol. 269, pp. 121–130, 2025, doi: 10.1016/j.procs.2025.08.265.
[8] A. AlMasaud and H. H. Al-Baity, “On the robustness of Arabic aspect-based sentiment analysis: A comprehensive exploration of transformer-based models,” J. King Saud Univ. Comput. Inf. Sci., vol. 36, no. 10, Art. no. 102264, Dec. 2024, doi: 10.1016/j.jksuci.2024.102264.
[9] S. Taj, S. M. Daudpota, A. S. Imran, and Z. Kastrati, “Aspect-based sentiment analysis for software requirements elicitation using fine-tuned Bidirectional Encoder Representations from Transformers and Explainable Artificial Intelligence,” Eng. Appl. Artif. Intell., vol. 151, Art. no. 110632, Jul. 2025, doi: 10.1016/j.engappai.2025.110632.
[10] F. Koto, J. H. Lau, and T. Baldwin, “IndoBERTweet: A pretrained language model for Indonesian Twitter with effective domain-specific vocabulary initialization,” in Proc. 2021 Conf. Empirical Methods Natural Language Processing (EMNLP), Online and Punta Cana, Dominican Republic, Nov. 2021, pp. 10660–10668, doi: 10.18653/v1/2021.emnlp-main.833.
[11] M. H. Popescu, K. Roitero, and V. Della Mea, “Integrating confidence, difficulty, and language model calibration for better explainability in clinical documents coding: Applications of AI,” JMIR AI, vol. 5, Art. no. e78764, Apr. 2026, doi: 10.2196/78764.
[12] J. Zhang, W. Yao, X. Chen, and L. Feng, “Transferable post-hoc calibration on pretrained transformers in noisy text classification,” Proc. AAAI Conf. Artif. Intell., vol. 37, no. 11, pp. 13940–13948, Jun. 2023, doi: 10.1609/aaai.v37i11.26632.
[13] D. Murthy et al., “Categorizing E-cigarette-related tweets using BERT topic modeling,” Emerg. Trends Drugs Addict. Health, vol. 4, Art. no. 100160, Dec. 2024, doi: 10.1016/j.etdah.2024.100160.
[14] C. C. Silva, M. Galster, and F. Gilson, “Applying short text topic models to instant messaging communication of software developers,” J. Syst. Softw., vol. 216, Art. no. 112111, Oct. 2024, doi: 10.1016/j.jss.2024.112111.
[15] M. Grootendorst, “BERTopic: Neural topic modeling with a class-based TF-IDF procedure,” arXiv:2203.05794, Mar. 2022.
[16] M. Grootendorst, “BERTopic,” BERTopic Documentation. Accessed: Jun. 4, 2026. [Online]. Available: BERTopic Documentation
[17] M. D. Angelo, R. W. Harwenda, I. Budi, A. B. Santoso, and P. K. Putra, “Sentiment analysis and topic modeling of public opinion on Indonesia new capital city development policies,” Eduvest J. Univ. Stud., vol. 5, no. 5, pp. 5951–5969, May 2025, doi: 10.59188/eduvest.v5i5.51234.
[18] C. Li and X. Hu, “Medical artificial intelligence in scholarly and public perspective: BERTopic-based analysis of topic-sentiment collaborative mining,” Data Sci. Informetr., vol. 5, no. 1, pp. 33–42, Mar. 2025, doi: 10.1016/j.dsim.2025.05.001.
[19] Selenium, “WebDriver,” Selenium Documentation. Accessed: Jun. 4, 2026. [Online]. Available: Selenium WebDriver Documentation
[20] A. S. Franzke, A. Bechmann, M. Zimmer, and C. M. Ess, Internet Research: Ethical Guidelines 3.0. Association of Internet Researchers, 2020.
[21] J. Taylor and C. Pagliari, “Mining social media data: How are research sponsors and researchers addressing the ethical challenges?” Res. Ethics, vol. 14, no. 2, pp. 1–39, Apr. 2018, doi: 10.1177/1747016117738559.
[22] J. Cohen, “A coefficient of agreement for nominal scales,” Educ. Psychol. Meas., vol. 20, no. 1, pp. 37–46, Apr. 1960, doi: 10.1177/001316446002000104.
[23] Scikit-learn Developers, “train_test_split,” scikit-learn 1.9.0 Documentation. Accessed: Jun. 4, 2026. [Online]. Available: scikit-learn train_test_split documentation
[24] S. Desai and G. Durrett, “Calibration of pre-trained transformers,” in Proc. 2020 Conf. Empirical Methods Natural Language Processing (EMNLP), Online, Nov. 2020, pp. 295–302, doi: 10.18653/v1/2020.emnlp-main.21.
[25] M. Minderer et al., “Revisiting the calibration of modern neural networks,” in Adv. Neural Inf. Process. Syst., vol. 34, pp. 15682–15694, 2021.
[26] C. Guo, G. Pleiss, Y. Sun, and K. Q. Weinberger, “On calibration of modern neural networks,” in Proc. 34th Int. Conf. Mach. Learn. (ICML), vol. 70, Sydney, NSW, Australia: PMLR, 2017, pp. 1321–1330.
[27] M. Chidambaram and R. Ge, “Reassessing how to compare and improve the calibration of machine learning models,” in Proc. Int. Conf. Learn. Represent. (ICLR), 2025.
[28] L. McInnes, J. Healy, N. Saul, and L. Großberger, “UMAP: Uniform manifold approximation and projection,” J. Open Source Softw., vol. 3, no. 29, Art. no. 861, Sep. 2018, doi: 10.21105/joss.00861.
[29] H. Rahimi, D. Mimno, J. L. Hoover, H. Naacke, C. Constantin, and B. Amann, “Contextualized topic coherence metrics,” in Findings Assoc. Comput. Linguistics: EACL 2024, St. Julian’s, Malta, Mar. 2024, pp. 1760–1773, doi: 10.18653/v1/2024.findings-eacl.123.
Downloads
Published
Issue
Section
License
Copyright (c) 2026 Journal of Information Systems and Informatics
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors Declaration
- The Authors certify that they have read, understood, and agreed to the Journal of Information Systems and Informatics (JournalISI) submission guidelines, policies, and submission declaration. The submission has been prepared using the provided template.
- The Authors certify that all authors have approved the publication of this manuscript and that there is no conflict of interest.
- The Authors confirm that the manuscript is their original work, has not received prior publication, is not under consideration for publication elsewhere, and has not been previously published.
- The Authors confirm that all authors listed on the title page have contributed significantly to the work, have read the manuscript, attest to the validity and legitimacy of the data and its interpretation, and agree to its submission.
- The Authors confirm that the manuscript is not copied from or plagiarized from any other published work.
- The Authors declare that the manuscript will not be submitted for publication in any other journal or magazine until a decision is made by the journal editors.
- If the manuscript is finally accepted for publication, the Authors confirm that they will either proceed with publication immediately or withdraw the manuscript in accordance with the journal’s withdrawal policies.
- The Authors agree that, upon publication of the manuscript in this journal, they transfer copyright or assign exclusive rights to the publisher, including commercial rights
