Reconstruction of Missing Meteorological Observation Data Using a Hybrid LSTM-KNN Model in a Web-Based Monitoring System
DOI:
https://doi.org/10.29303/goescienceed.v7i4.3215Keywords:
AWS, Missing Data, Hybrid LSTM-KNN, Data Reconstruction, Dynamic Weighted Blending.Abstract
Automatic Weather Stations frequently lose data because of sensor faults, power failures, or network disruption, which lowers the validity of observational records. This study develops a hybrid Long Short-Term Memory (LSTM) and K-Nearest Neighbors (KNN) model to reconstruct missing data and implements it in a web-based system. KNN acts as the initial estimator based on seasonal analogues, while LSTM refines the temporal pattern. Both estimates are merged through a dynamic weighted blending mechanism that decays with gap duration to prevent autoregressive drift, complemented by buffer isolation so the LSTM never reads its own predictions as context. The system was built with Streamlit and SQLite, covering layered quality control, reconstruction, and a monitoring dashboard. Evaluation used nine artificial gap scenarios on AWS data from Tanjung Priok Station for 2020 to 2026, covering air temperature, relative humidity, and air pressure. The hybrid advantage proved conditional on gap structure and duration. On block gaps it lowered MAE by up to 2.66% over KNN-only, peaking at ten minutes and shrinking to 0.07% at six hours, whereas on sporadic MCAR gaps it gave no advantage. The findings show that adaptive weight decay is essential in operational systems.
References
Afrifa-Yamoah, E., Mueller, U. A., Taylor, S. M., & Fisher, A. J. (2020). Missing data imputation of high-resolution temporal climate time series data. Meteorological Applications, 27(1). https://doi.org/10.1002/met.1873
Ahn, H., Sun, K., & Kim, K. P. (2022). Comparison of missing data imputation methods in time series forecasting. Computers, Materials & Continua, 70(1), 767-779. https://doi.org/10.32604/cmc.2022.019369
Akkem, Y., Biswas, S. K., & Varanasi, A. (2024). Streamlit-based enhancing crop recommendation systems with advanced explainable artificial intelligence for smart farming. Neural Computing and Applications. https://doi.org/10.1007/s00521-024-10208-z
Almeida, A., Bras, S., Sargento, S., & Pinto, F. C. (2024). Focalize K-NN: An imputation algorithm for time series datasets. Pattern Analysis and Applications, 27(2). https://doi.org/10.1007/s10044-024-01262-3
Boomgard-Zagrodnik, J. P., & Brown, D. J. (2022). Machine learning imputation of missing Mesonet temperature observations. Computers and Electronics in Agriculture, 192, 106580. https://doi.org/10.1016/j.compag.2021.106580
Cerqueira, V., & Torgo, L. (2023). Multi-output ensembles for multi-step forecasting (arXiv:2306.14563). arXiv. https://arxiv.org/abs/2306.14563
Ehsan, A., Abuhaliqa, M. A. M. E., Catal, C., & Mishra, D. (2022). RESTful API testing methodologies: Rationale, challenges, and solution directions. Applied Sciences, 12(9), 4369. https://doi.org/10.3390/app12094369
Gaffney, K. P., Prammer, M., Brasfield, L., Hipp, D. R., Kennedy, D., & Patel, J. M. (2022). SQLite: Past, present, and future. Proceedings of the VLDB Endowment, 15(12), 3535-3547. https://doi.org/10.14778/3554821.3554842
Hyndman, R. J., & Athanasopoulos, G. (2021). Forecasting: Principles and practice (3rd ed.). OTexts. https://otexts.com/fpp3/
Laeeq, A., Li, J., & Adeel, U. (2026). A hybrid LSTM-attention model for multivariate time series imputation: Evaluation on environmental datasets. Machine Learning and Knowledge Extraction, 8(1), 18. https://doi.org/10.3390/make8010018
Lattifia, T., Buana, P. W., & Rusjayanthi, N. K. D. (2022). Model prediksi cuaca menggunakan metode LSTM. JITTER: Jurnal Ilmiah Teknologi dan Komputer, 3(1), 994-1002. https://doi.org/10.24843/jtrti.2022.v03.i01.p35
Liu, M., Huang, Y., Li, Z., Tong, B., Liu, Z., Sun, M., Jiang, F., & Zhang, H. (2020). The applicability of LSTM-KNN model for real-time flood forecasting in different climate zones in China. Water, 12(2), 440. https://doi.org/10.3390/w12020440
Luo, X., Li, D., Yang, Y., & Zhang, S. (2019). Spatiotemporal traffic flow prediction with KNN and LSTM. Journal of Advanced Transportation, 2019, 1-10. https://doi.org/10.1155/2019/4145353
Nida, H., Kashif, M., Khan, M. I., & Ghamkhar, M. (2023). Comparison of missing data imputation methods using weather data. Pakistan Journal of Agricultural Sciences, 60(2), 327-336. https://doi.org/10.21162/PAKJAS/23.228
Nugraha, Y. E. N., Ariawan, I., & Arifin, W. A. (2023). Weather forecast from time series data using LSTM algorithm. Jurnal Teknologi Informasi dan Komunikasi, 14(1), 144-152. https://doi.org/10.51903/jtikp.v14i1.531
Nugroho, A. A., & Haris, M. (2024). Analisis efektivitas teknik imputasi pada LSTM untuk meningkatkan kualitas data pada peramalan curah hujan. Jurnal Informatika dan Rekayasa Elektronik, 7(2), 301-311. https://doi.org/10.36595/jire.v7i2.1172
Oktaviani, I., & Putrada, A. (2022). KNN imputation to missing values of regression-based rain duration prediction on BMKG data. Jurnal Infotel, 14(4), 249-254. https://doi.org/10.20895/infotel.v14i4.840
Ouyang, Z., Ravier, P., & Jabloun, M. (2022). Are deep learning models practically good as promised? A strategic comparison of deep learning models for time series forecasting. In Proceedings of the 2022 European Signal Processing Conference (EUSIPCO) (pp. 1477–1481). https://eurasip.org/Proceedings/Eusipco/Eusipco2022/pdfs/0001477.pdf
Purwandari, K., Aufauzan, B. T., & Sigalingging, J. W. C. (2026). Web-based quality control dashboard design for data validation and monitoring: A case study of BMKG instruments. Engineering, Mathematics and Computer Science Journal, 8(1).
Sugiarto, B., Rahmat, R. F., & Purnamawati, S. (2019). Rancang bangun Automatic Weather Station berbasis mikrokontroler. Jurnal Teknologi Informasi dan Ilmu Komputer, 6(3), 245-252.
Teutsch, P., & Mäder, P. (2022). Flipped classroom: Effective teaching for time series forecasting. Transactions on Machine Learning Research. https://openreview.net/forum?id=w3x20YEcQK
Wahyudi, A., & Widyarto, W. O. (2025). Deteksi anomali dan rekonstruksi data pengamatan otomatis berbasis machine learning. Jurnal Instrumentasi dan Teknologi Meteorologi, 8(1), 33-44.
World Meteorological Organization. (2023). Guide to instruments and methods of observation (WMO-No. 8, Vol. V).
Yang, L., & Shami, A. (2020). On hyperparameter optimization of machine learning algorithms: Theory and practice. Neurocomputing, 415, 295-316. https://doi.org/10.1016/j.neucom.2020.07.061.
