对于皮带廊的工业物联网大数据分析和可视化系统的设计与实现

       皮带传输廊道由于距离长且需要连续长时间运行，布置有丰富的传感器来监 视产线运行情况，监测数据会随着时间的推移大量累积。这些数据中含有丰富的 有价值的信息可以挖掘。但是其多样的数据类型与大数据量是现在工厂信息检索 分析系统无法胜任的。目前常见的大数据处理技术多来源于互联网领域，但是工 业物联网中的数据有其自身独有的依从时序、类型多样、来源灵活等特点，使得 互联网大数据的处理方法在此处不能完全适用。针对以上问题，本文构建了适用 于工业物联网场景的大数据平台，利用非结构型的数据库和分布式存储计算技术 来支撑大规模、时序、多类型数据的存储与快速查找分析。在边缘侧传感器算法 中，提出并实现了基于内存数据库 Redis 的存储系统，将原来只存储在内存中的数 据分担到硬盘中，使得数据格式更加清晰，边缘侧的计算资源利用更加充分。整个 平台系统可以分为四层。首先是数据采集层，将来自传感器的数据进行汇总，重新 构建数据格式；然后是数据存储层，基于 Hadoop 构建了分布式存储集群，数据分 散存储在可以扩展的分布式文件系统里和非结构型数据库 Redis 中；之后是数据分 析层，使用了分布式计算框架 Spark，以便在分布式的计算机集群中做到对于大数 据的快速查询与分析；最上层是数据可视化层，使用了轻量级 Web 网站框架 Flask 构建可视化界面，对传感器监测的工况进行展示，支持历史数据查询，并进行统计 操作，然后将结果数据通过可视化工具 Echarts 展示在面板上。经过测试，整个系 统对皮带廊场景的数据任务能够良好的支持，未来可以推广应用到其它工业物联 网场景。

Due to the long distance and the continuous long-term operation of the belt conveyor corridor, a wealth of sensors are arranged to monitor the operation of the production line, and the data will accumulate a lot over time. The data contains a wealth of valuable information that can be mined. However, its diverse data types and large data volume are beyond the capabilities of current factory information retrieval and analysis systems. At present, most of the big data processing technologies come from the Internet industry. However, the data in the Industrial Internet of Things has its own characteristics such as time-series, diverse types, and flexible sources, which make the Internet big data processing methods not fully applicable here. To solve the above problems, this paper builds a big data platform suitable for industrial IoT scenarios, using unstructured databases and distributed storage computing technology to support storage, fast search and analysis for large-scale, time-series, and multi-type data. In the sensor algorithm on the edge side, a storage system based on the memory database Redis is proposed and implemented, which shares the data originally only stored in the memory to the hard disk, which makes the data format clearer and the computing resources on the edge side are more fully utilized. The entire platform system can be divided into four layers. The first is the data acquisition layer, which aggregates the data from the sensors and reconstructs the data format; the second is the data storage layer, which builds a distributed storage cluster based on Hadoop and the database Redis; followed by the data analysis layer, using the distributed computing framework Spark, in order to achieve fast query and analysis of big data in distributed computer clusters; the top layer is the data visualization layer, using lightweight website framework Flask builds a visual interface, displays the working conditions monitored by the sensor, supports historical data query, and performs statistical operations, and then displays the result data on the panel through the visualization tool Echarts. After testing, the whole system can well support the data tasks of the belt corridor, and can be applied to other industrial IoT scenarios in the future.

[1] WHITE T. Hadoop: The Definitive Guide[J]. O’rlly Media Inc Gravenstn Highway North,2012, 215(11): 1 - 4.
[2] GHEMAWAT S, GOBIOFF H, LEUNG S T. The Google file system[J]. Acm Sigops OperatingSystems Review, 2003, 37(5): 29-43.
[3] DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters[J]. Com munications of the ACM, 2008, 51(1): 107-113.
[4] 史晓丽. Bigtable 分布式存储系统的研究[D]. 西安电子科技大学.
[5] CAFARELLA M, CUTTING D. Building Nutch: Open Source Search: A case study in writingan open source search engine[J]. Queue, 2004, 2(2): 54-61.
[6] KARAU H, KONWINSKI A, WENDELL P, et al. Learning spark: lightning-fast big dataanalysis[M]. ” O’Reilly Media, Inc.”, 2015.
[7] 赵晟, 姜进磊. 典型大数据计算框架分析[J]. 中兴通讯技术, 2016, 22(2): 5.
[8] 徐涛. 结构化大数据存储与查询优化关键技术[D]. 清华大学.
[9] HAI R, GEISLER S, QUIX C. Constance: An Intelligent Data Lake System[C/OL]//SIGMOD’16: Proceedings of the 2016 International Conference on Management of Data. New York,NY, USA: Association for Computing Machinery, 2016: 2097–2100. https://doi.org/10.1145/2882903.2899389.
[10] 阿里巴巴. 阿里云数据中台[EB/OL]. (2020-09-17)
[2022-04-20]. https://dp.alibaba.com/exchange/article/145?spm=a215hz.13439232.0.0.5aa67ef85sP43K.
[11] GARTNER. Big Data[EB/OL]. (2022-04-20)
[2022-04-20]. https://www.gartner.com/en/information-technology/glossary/big-data.
[12] MEIER A, KAUFMANN M. SQL & NoSQL Databases: models, languages, consistencyoptions and architectures for Big Data Management[M]. Wiesbaden [Heidelberg]: SpringerVieweg, 2019.
[13] FOR ECONOMICS AFFAIRS F M, ACTION C. Industrie 4.0[J/OL]. BMWK - Industrie 4.0.https://www.bmwi.de/Redaktion/EN/Dossier/industrie-40.html.
[14] OF ECONOMY M, FINANCE. New Industrial France[EB/OL]. (2015-10-1)
[2022-04-20].https://www.economie.gouv.fr/files/files/PDF/web-dp-indus-ang.pdf.
[15] NIST OFFICE OF ADVANCED MANUFACTURING. National Strategic Plan for AdvancedManufacturing[EB/OL]. (2021-12-17)
[2022-04-20]. https://www.nist.gov/oam/national-strategic-plan-advanced-manufacturing.
[16] BOYES H, HALLAQ B, CUNNINGHAM J, et al. The industrial internet of things (IIoT): Ananalysis framework[J/OL]. Computers in Industry, 2018, 101: 1-12. https://www.sciencedirect.com/science/article/pii/S0166361517307285. DOI: https://doi.org/10.1016/j.compind.2018.04.015.
[17] KHAN W, REHMAN M, ZANGOTI H, et al. Industrial internet of things: Recent advances,enabling technologies and open challenges[J/OL]. Computers & Electrical Engineering, 2020,81: 106522. https://www.sciencedirect.com/science/article/pii/S0045790618329550. DOI:https://doi.org/10.1016/j.compeleceng.2019.106522.
[18] LEE J, STANLEY M, SPANIAS A, et al. Integrating machine learning in embedded sensor sys tems for Internet-of-Things applications[C/OL]//2016 IEEE International Symposium on SignalProcessing and Information Technology (ISSPIT). 2016: 290-294. DOI: 10.1109/ISSPIT.2016.7886051.
[19] COROTINSCHI G, GăITAN V G. The development of IoT applications using old hardwareequipment and virtual TEDS[C/OL]//2016 International Conference on Development and Ap plication Systems (DAS). 2016: 264-268. DOI: 10.1109/DAAS.2016.7492584.
[20] TEPIć S, PEJIć P, DOMšIć J, et al. IBMS - Intelligent Building Management System Framework[C/OL]//2015 38th International Convention on Information and Communication Technology,Electronics and Microelectronics (MIPRO). 2015: 143-148. DOI: 10.1109/MIPRO.2015.7160254.
[21] LEE I, LEE K. The Internet of Things (IoT): Applications, investments, and challenges forenterprises[J/OL]. Business Horizons, 2015, 58(4): 431-440. https://www.sciencedirect.com/science/article/pii/S0007681315000373. DOI: https://doi.org/10.1016/j.bushor.2015.03.008.
[22] TAO F, WANG Y, ZUO Y, et al. Internet of Things in product life-cycle energy management[J/OL]. Journal of Industrial Information Integration, 2016, 1: 26-39. https://www.sciencedirect.com/science/article/pii/S2452414X16000030. DOI: https://doi.org/10.1016/j.jii.2016.03.001.
[23] MAGNO M, SPADARO L, SINGH J, et al. Kinetic energy harvesting: Toward autonomouswearable sensing for Internet of Things[C/OL]//2016 International Symposium on Power Elec tronics, Electrical Drives, Automation and Motion (SPEEDAM). 2016: 248-254. DOI:10.1109/SPEEDAM.2016.7525995.
[24] LIU M, YU F R, TENG Y, et al. Performance Optimization for Blockchain-Enabled IndustrialInternet of Things (IIoT) Systems: A Deep Reinforcement Learning Approach[J/OL]. IEEETransactions on Industrial Informatics, 2019, 15(6): 3559-3570. DOI: 10.1109/TII.2019.2897805.
[25] HALPIN H, MONNIN A. Interview with Tim Berners-Lee[M]. Philosophical Engineering:Toward a Philosophy of the Web, 2014.
[26] O’REILLY T. What is Web 2.0: Design patterns and business models for the next generationof software[J]. Communications & strategies, 2007(1): 17.
[27] 蒋昆佑. 基于 Spark 的海量数据计算平台设计与实现[D]. 大连理工大学, 2016.
[28] FRIENDLY M, DENIS D J. Milestones in the history of thematic cartography, statistical graphics, and data visualization[J]. URL http://www. datavis. ca/milestones, 2001, 32: 13.
[29] PLAYFAIR W. The Commercial and Political Atlas, Representing, by means of Stained Copper Plate Charts, The Progress of the Commerce, Revenues, Expenditure, and Debts of England,during the whole of the Eighteenth Century[J]. Bmj Case Reports, 2007, 2013(10): 1245-1253.
[30] AZZAM T, EVERGREEN S, GERMUTH A A, et al. Data Visualization and Evaluation[J].New Directions for Evaluation, 2013, 2013(139): 7-32.
[31] GUIVANT, J. E, NEBOT, et al. Solving computational and memory requirements of feature based simultaneous localization and mapping algorithms[J]. Robotics and Automation, IEEETransactions on, 2003, 19(4): 749-755.
[32] BRESSON G, FERAUD T, AUFRERE R, et al. Real-Time Monocular SLAM With Low Mem ory Requirements[J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 16(4):1827-1839.
[33] LABBé M, MICHAUD F. Memory management for real-time appearance-based loop clo sure detection[C]//2011 IEEE/RSJ International Conference on Intelligent Robots and Systems,IROS 2011, San Francisco, CA, USA, September 25-30, 2011. 2011.
[34] LABBé M, MICHAUD F. Long-term online multi-session graph-based SPLAM with memorymanagement[J]. Autonomous Robots, 2017.
[35] LABBE M, MICHAUD F. Appearance-Based Loop Closure Detection for Online Large-Scaleand Long-Term Operation[J]. IEEE Transactions on Robotics, 2013, 29(3): 734-745.
[36] LABBE M, MICHAUD F. Online global loop closure detection for large-scale multi-sessiongraph-based SLAM[C]//2014 IEEE/RSJ International Conference on Intelligent Robots andSystems. 2014.
[37] CANOVAS B, ROMBAUT M, NEGRE A, et al. Speed and Memory Efficient Dense RGB-DSLAM in Dynamic Scenes[C]//2020 IEEE/RSJ International Conference on Intelligent Robotsand Systems (IROS). 2020.
[38] KRETZSCHMAR H, GRISETTI G, STACHNISS C. Lifelong Map Learning for Graph-basedSLAM inStaticEnvironments[J]. Ki Künstliche Intelligenz, 2010, 24(3): 199-206.
[39] GALVEZ-LPEZ D, TARDOS J D. Bags of Binary Words for Fast Place Recognition in ImageSequences[J]. IEEE Transactions on Robotics, 2012, 28(5): 1188-1197.
[40] GARCIA-FIDALGO E, ORTIZ A. iBoW-LCD: An Appearance-based Loop Closure DetectionApproach using Incremental Bags of Binary Words[J]. IEEE Robotics and Automation Letters,2018, 3(4): 3051-3057.
[41] NEWMAN P, HO K. SLAM-Loop Closing with Visually Salient Features[C]//IEEE Interna tional Conference on Robotics & Automation. 2005.
[42] BEN ALI A J, HASHEMIFAR Z S, DANTU K. Edge-SLAM: edge-assisted visual simultane ous localization and mapping[M]//Proceedings of the 18th International Conference on MobileSystems, Applications, and Services. 2020: 325-337.
[43] ZHANG Y, HSIAO M, ZHAO Y, et al. Distributed Client-Server Optimization for SLAM withLimited On-Device Resources[Z]. 2021.
[44] BENAVIDEZ P, MUPPIDI M, RAD P, et al. Cloud-based realtime robotic Visual SLAM[J].IEEE, 2015.
[45] FU L, YANG S, YI X, et al. Towards Visual SLAM with Memory Management for Large-scaleEnvironments[M]. Springer, Cham, 2017.
[46] SUGER B, TIPALDI G D, SPINELLO L, et al. An approach to solving large-scale SLAMproblems with a small memory footprint[C]//2014 IEEE International Conference on Roboticsand Automation (ICRA). 2014.
[47] JO H G, CHO H M, JO S, et al. Efficient Grid-Based Rao-Blackwellized Particle Filter SLAMWith Inter-Particle Map Sharing[J]. IEEE/ASME Transactions on Mechatronics, 2018: 1-1.
[48] MüHLFELLNER P, FURGALE P, DERENDARZ W, et al. Designing a Relational Databasefor Long-Term Visual Mapping[Z]. 2015.
[49] FOURIE D, CLAASSENS S, PILLAI S, et al. SLAMinDB: Centralized graph databases formobile robotics[C]//2017 IEEE International Conference on Robotics and Automation (ICRA).2017.
[50] DOAN A D, LATIF Y, CHIN T J, et al. HM4: Hidden Markov Model with Memory Manage ment for Visual Place Recognition[Z]. 2020.
[51] OUYANG M, SHI X, WANG Y, et al. A Collaborative Visual SLAM Framework for ServiceRobots[Z]. 2021.
[52] DB-ENGINES. DB-Engines Ranking of Key-value Stores[EB/OL]. (2022-04-20)
[2022-04-20].https://db-engines.com/en/ranking/key-value+store.
[53] GESSERT F, WINGERATH W, FRIEDRICH S, et al. NoSQL database systems: a survey anddecision guidance[J]. Computer Science Research & Development, 2017.
[54] RUNOOB.COM. Redis 与其他 key-value 存储有什么不同[EB/OL]. (2022-04-20)
[2022-04-20]. https://www.runoob.com/redis/redis-intro.html.
[55] 曾刚. 实战 Hadoop 大数据处理[M]. 实战 Hadoop 大数据处理, 2015.
[56] ZAHARIA M, CHOWDHURY M, FRANKLIN M J, et al. Spark: Cluster computing withworking sets[C]//2nd USENIX Workshop on Hot Topics in Cloud Computing (HotCloud 10).2010.
[57] LI D, MEI H, SHEN Y, et al. ECharts: A declarative framework for rapid construction ofweb-based visualization[J]. Visual Informatics, 2018, 2: 136-146.
[58] MUR-ARTAL R, TARDóS J. ORB-SLAM2: An Open-Source SLAM System for Monocular,Stereo, and RGB-D Cameras[J]. IEEE Transactions on Robotics, 2017.
[59] GALVEZ-LPEZ D, TARDOS J D. Bags of Binary Words for Fast Place Recognition in ImageSequences[J]. IEEE Transactions on Robotics, 2012, 28(5): 1188-1197.
[60] SAMMUT C, WEBB G I. TF–IDF[M/OL]. Boston, MA: Springer US, 2010: 986-987. https://doi.org/10.1007/978-0-387-30164-8_832.
[61] MUR-ARTAL M J M M, Raúl, TARDÓS J D. ORB-SLAM: a Versatile and Accurate MonocularSLAM System[J/OL]. IEEE Transactions on Robotics, 2015, 31(5): 1147-1163. DOI: 10.1109/TRO.2015.2463671.