Automatic Quality Control of Crowdsourced Rainfall Data With Multiple Noises: A Machine Learning Approach

Niu, Geng1,2,3; Yang, Pan3; Zheng, Yi2,4; Cai, Ximing3; Qin, Huapeng1

2021-11-01

10.1029/2020WR029121

WATER RESOURCES RESEARCH   影响因子和分区

0043-1397

1944-7973

In geophysics, crowdsourcing is an emerging nontraditional environmental monitoring approach that supports data acquisition from individual citizens. However, because of the involvement of undertrained citizens and imprecise low-cost sensors, crowdsourced data applications suffer from different types of noises that can deteriorate the overall monitoring accuracy. In this study, we propose a machine learning approach for automatic crowdsourced data quality control (CSQC) that detects and removes noisy data inputs in spatially and temporally discrete crowdsourced observations coming from both fixed-point sensors (e.g., surveillance cameras) and moving sensors (e.g., moving cars/pedestrians). We design a set of features from original and interpolated rainfall data and use them to train and test the CSQC models using both supervised and unsupervised machine learning algorithms. The performances of the CSQC models under various scenarios assuming no retraining are also tested (hereafter referred to as transferability). The results based on synthetic but realistic data show that the CSQC models can significantly reduce the overall rainfall estimate errors. Under the stationary assumption, the CSQC models based on both supervised and unsupervised algorithms perform well in noisy data identification and overall rainfall estimation error reduction; however, if the model is transferred to other cities with different rainfall patterns or noise compositions (without retraining), supervised multilayer perceptrons (MLPs) show the best performance.

crowdsourcing rainfall machine learning quality control transferability

SCI ; EI

英语

通讯

National Science Foundation of China[51961125203] ; Shenzhen Science and Technology Innovation Commission[KCXFZ202002011006491] ; China Scholarship Council[201806010249]

Environmental Sciences & Ecology ; Marine & Freshwater Biology ; Water Resources

Environmental Sciences ; Limnology ; Water Resources

WOS:000723106900040

AMER GEOPHYSICAL UNION

20214811239557

Data acquisition ; Learning algorithms ; Machine learning ; Monitoring ; Quality control ; Rain ; Security systems

Precipitation:443.3 ; Data Processing and Image Processing:723.2 ; Machine Learning:723.4.2 ; Quality Assurance and Control:913.3 ; Accidents and Accident Prevention:914.1

ENVIRONMENT/ECOLOGY

Web of Science

1.Peking Univ, Key Lab Urban Habitat Environm Sci & Technol, Shenzhen Grad Sch, Sch Environm & Energy, Shenzhen, Peoples R China
2.Southern Univ Sci & Technol, Sch Environm Sci & Engn, Shenzhen, Peoples R China
3.Univ Illinois, Dept Civil & Environm Engn, Champaign, IL 61820 USA
4.Southern Univ Sci & Technol, Shenzhen Municipal Engn Lab Environm IoT Technol, Shenzhen, Peoples R China

Niu, Geng,Yang, Pan,Zheng, Yi,et al. Automatic Quality Control of Crowdsourced Rainfall Data With Multiple Noises: A Machine Learning Approach[J]. WATER RESOURCES RESEARCH,2021,57(11).

Niu, Geng,Yang, Pan,Zheng, Yi,Cai, Ximing,&Qin, Huapeng.(2021).Automatic Quality Control of Crowdsourced Rainfall Data With Multiple Noises: A Machine Learning Approach.WATER RESOURCES RESEARCH,57(11).

Niu, Geng,et al."Automatic Quality Control of Crowdsourced Rainfall Data With Multiple Noises: A Machine Learning Approach".WATER RESOURCES RESEARCH 57.11(2021).