基于强化对抗模型的开放式工业控制系统内生功能安全研究

随着工业控制系统中信息技术与操作技术的逐步融合，系统获得了前所未有的效率提升和灵活性，但也面临着日益严峻的网络安全威胁。著名的Stuxnet恶意软件针对控制系统中可编程逻辑控制器的设定值进行攻击，直接导致了系统功能失效，造成了严重后果，这凸显了现代网络威胁的复杂性以及保障功能安全对关键基础设施系统安全运行的重要性。面对不断演变的网络安全威胁，加强功能安全防护已成为确保系统可靠稳定运行的当务之急。内生安全理念的提出为解决这一难题提供了新的视角。该理念强调系统通过自身的特性和机制实现主动安全防护，摆脱了对外部防御措施的依赖。基于此视角，根据工业控制系统行为是否可预料、是否安全性以及是否经济分为五种类型。本文重点关注预料之外的异常行为（UA）和预料之外的正常且经济的行为（UNE）这两类行为。UA行为是功能安全面临的主要挑战，也是内生安全理念应对的核心问题。而发掘UNE行为则可以在保障安全的前提下，优化系统运行、提升经济效益，实现安全与效益的双赢。
鉴于设定点对工业控制系统运行的重要影响，本文聚焦于设定点误操作或恶意攻击引发的功能安全问题，创新性地提出了一种融合深度强化学习技术和内生安全理念的强化对抗模型。在该模型中，强化学习智能体通过持续与环境交互，不断丰富完善行为规则模型。行为规则模型不仅表征已知行为模式，更作为先验知识指导智能体避免重复探索，从而推动发现更多未知的预料之外行为，实现“未知的未知”到“已知”的转化。模型采用近端策略优化算法，结合高斯分布和贝塔分布对智能体的策略进行训练和优化。
为验证所提出方法的有效性，本文选取经典的田纳西伊斯曼工业过程作为案例研究，详细展示了将所提出的强化对抗模型框架应用在具体工业过程上的细节设定。实验结果表明，对于预料之外异常行为的探索，强化对抗模型相比传统方法表现出更高效的探索能力。此外，模型还展现出优化系统运行和提升经济效益的能力，在保障安全的同时实现了效益优化。这些结果证实了将深度强化学习与内生安全理念相结合的可行性和优越性，为解决工业控制系统的功能安全难题提供了新的思路和方法。

[1] LANGNER R. Stuxnet: Dissecting a cyberwarfare weapon[J]. IEEE Security & Privacy, 2011, 9(3): 49-51.
[2] 工业和信息化部网站. 《工业互联网创新发展行动计划（2021-2023 年）》解读[EB/OL]. http://www.gov.cn/zhengce/2021-02/18/content_5587565.htm.
[3] IEC 61508: Functional safety of electrical/electronic/programmable electronic safety-related systems[S/OL]. International Electrotechnical Commission, 1998. https://law.resource.org/p ub/in/bis/S05/is.iec.61508.1.1998.pdf.
[4] WU J. Cyberspace Endogenous Safety and Security[J/OL]. Engineering, 2022, 15. https: //doi.org/10.1016/j.eng.2021.05.015.
[5] WU J. Development paradigms of cyberspace endogenous safety and security[J/OL]. Scientia Sinica Informationis, 2022, 52(2): 189-204. DOI: 10.1360/SSI-2021-0272.
[6] XIN Y. Protection Architecture of Endogenous Safety and Security for Industrial Control Sys- tems[J/OL]. Security and Safety, 2023, 2. DOI: 10.1051/sands/2023001.
[7] JIN L, HU X, WU J. From Perfect Secrecy to Perfect Safety and Security : Cryptography-Based Analysis of Endogenous Security: Vol. 2[EB/OL]. 2023. DOI: 10.1051/sands/2023004.
[8] LOPEZ-MARTIN M, CARRO B, SANCHEZ-ESGUEVILLAS A. Application of deep rein- forcement learning to intrusion detection for supervised problems[J/OL]. Expert Systems with Applications, 2020, 141(Ml): 1-24. DOI: 10.1016/j.eswa.2019.112963.
[9] SETHI K, KUMAR R, PRAJAPATI N, et al. Deep Reinforcement Learning based Intrusion Detection System for Cloud Infrastructure[J/OL]. 2020 International Conference on COMmu- nication Systems and NETworkS, COMSNETS 2020, 2020: 1-6. DOI: 10.1109/COMSNETS 48256.2020.9027452.
[10] LIU Z, WANG C, WANG W. Online Cyber-Attack Detection in the Industrial Control System: A Deep Reinforcement Learning Approach[J]. Mathematical Problems in Engineering, 2022, 2022.
[11] HUANG X, YUAN T, QIAO G, et al. Deep Reinforcement Learning for Multimedia Traffic Control in Software Defined Networking[J/OL]. IEEE Network, 2018, 32(6): 35-41. DOI: 10.1109/MNET.2018.1800097.
[12] LEI K, LIANG Y, LI W. Congestion control in SDN-based networks via multi-task deep rein- forcement learning[J/OL]. IEEE Network, 2020, 34(4): 28-34. DOI: 10.1109/MNET.011.190 0408.
[13] 朱愉田, 李华强. 基于深度强化学习的孤岛微电网故障区域判定[J]. 计算机仿真, 2021, 38(07): 78-82.
[14] DING Y, MA L, MA J, et al. Intelligent fault diagnosis for rotating machinery using deep Q- network based health state classification: A deep reinforcement learning approach[J]. Advanced Engineering Informatics, 2019, 42: 100977.
[15] FAN S, ZHANG X, SONG Z. Imbalanced sampleselection with deep reinforcement learning for fault diagnosis[J]. IEEE Transactions on Industrial Informatics, 2021, 18(4): 2518-2527.
[16] LIU X, OSPINA J, KONSTANTINOU C. Deep Reinforcement Learning for Cybersecurity Assessment of Wind Integrated Power Systems[J/OL]. IEEE Access, 2020, 8: 208378-208394. DOI: 10.1109/ACCESS.2020.3038769.
[17] PARK J, KIM T, SEONG S, et al. Control automation in the heat-up mode of a nuclear power plant using reinforcement learning[J]. Progress in Nuclear Energy, 2022, 145: 104107.
[18] ZHOU Y, ZHANG B, XU C, et al. A data-driven method for fast ac optimal power flow solutions via deep reinforcement learning[J]. Journal of Modern Power Systems and Clean Energy, 2020, 8(6): 1128-1139.
[19] STOUFFERK, FALCO J, SCARFONE K. Guide to Industrial Control Systems ( ICS ) Security Recommendations of the National Institute of Standards and Technology[J/OL]. NIST Special Publication, 2007, 2: 1-157. http://industryconsulting.org/pdfFiles/NISTDraft-SP800-82.pdf.
[20] BASHA O, ALAJMY J, NEWAZ T. Bhopal gas Tragedy: A safety case study[J]. The OAK- Trust Digital Repository, 2009.
[21] JAIN P, PASMAN H J, WALDRAM S P, et al. Did we learn about risk control since Seveso? Yes, we surely did, but is it enough? An historical brief and problem analysis[J/OL]. Journal of Loss Prevention in the Process Industries, 2017, 49: 5-17. http://dx.doi.org/10.1016/j.jlp.2 016.09.023.
[22] M.PIETERSEN C. The two largest industrial disasters, 25 year later The investigation, the facts and the importance for industrial safety[J/OL]. 13th Loss Prevention Conference, 2010 (December). https://www.researchgate.net/publication/321938846_The_two_largest_industrial_disasters_25_year_later_The_investigation_the_facts_and_the_importance_for_industrial_ safety.
[23] LEEMANN J E. Applying interactive planning at Dupont: The case of transforming a safety, health, and environmental function to deliver business value[J/OL]. Systemic Practice and Action Research, 2002, 15(2): 85-109. DOI: 10.1023/A:1015236423688.
[24] TANAKA H, FAN L T, LAIF S, et al. Fault-Tree Analysis By Fuzzy Probability.[J/OL]. IEEE Transactions on Reliability, 1983, R-32(5): 453-457. DOI: 10.1109/TR.1983.5221727.
[25] AKHTAR I, KIRMANI S. An Application of Fuzzy Fault Tree Analysis for Reliability Evaluation of Wind Energy System[J/OL]. IETE Journal of Research, 2020: 1-14. DOI: 10.1080/03772063.2020.1791741.
[26] TABESH M, ROOZBAHANI A, HADIGOL F, et al. Risk Assessment of Water Treatment Plants Using Fuzzy Fault Tree Analysis and Monte Carlo Simulation[J/OL]. Iranian Journal of Science and Technology - Transactions of Civil Engineering, 2022, 46(1): 643-658. https://doi.org/10.1007/s40996-020-00498-3.
[27] 姚海燕, 李勋章, 杨秀芹. 模糊故障树理论在航空充电设备故障诊断中的应用研究[J/OL]. 计量与测试技术, 2021, 48: 82-84. DOI: 10.15988/j.cnki.1004-6941.2021.6.026.
[28] MODARRES M, KAMINSKIY MP, KRIVTSOV V. Reliability engineering and risk analysis: a practical guide[M]. CRC press, 2016.
[29] DISTEFANO S, PULIAFITO A. Dynamic reliability block diagrams: Overview of a method- ology[J]. Proceedings of the European Safety and Reliability Conference 2007, ESREL 2007 - Risk,Reliability and Societal Safety, 2007, 2(January 2007): 1059-1068.
[30] ELDERHALLI Y, HASAN O, TAHAR S. A Formally Verified Algebraic Approach for Dy- namic Reliability Block Diagrams: August[R/OL]//Lecture Notes in Computer Science (in- cluding subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2019: 253-269. DOI: 10.1007/978-3-030-32409-4_16.
[31] XU H, XING L, ROBIDOUX R. Drbd: Dynamic reliability block diagrams for system relia- bility modelling[J]. International journal of computers and applications, 2009, 31(2): 132-141.
[32] KOVTUN V, IZONIN I, GREGUS M. The functional safety assessment of cyber-physical system operation process described by Markov chain[J/OL]. Scientific Reports, 2022, 12(1): 1-13. https://doi.org/10.1038/s41598-022-11193-w.
[33] BHATTI Z E, ROOP P S, SINHA R. Unified Functional Safety Assessment of Industrial Au- tomation Systems[J/OL]. IEEE Transactions on Industrial Informatics, 2017, 13(1): 17-26. DOI: 10.1109/TII.2016.2610185.
[34] ОдарущенкоОлегМиколайович, ОдарущенкоОленаБорисівна, ХарченкоВячеславСергійович . MARKOV MODELS FOR FUNCTIONAL SAFETY ASSESSMENT OF INSTRUMENTATION AND CONTROL SYSTEMS BASED ON SELF-CHECKING PROGRAMMABLE PLATFORMS[J]. Radioelectronic and Computer Systems, 2019(4): 17-29.
[35] NIESEN T, HOUY C, FETTKE P, et al. Towards an integrative big data analysis framework for data-driven risk management in industry 4.0[J/OL]. Proceedings of the Annual Hawaii International Conference on System Sciences, 2016, 2016-March: 5065-5074. DOI: 10.1109/ HICSS.2016.627.
[36] PALTRINIERIN, COMFORT L, RENIERS G. Learning about risk: Machine learning for risk assessment[J/OL]. Safety Science, 2019, 118(May): 475-486. https://doi.org/10.1016/j.ssci.2 019.06.001.
[37] JAMSHIDI A, FAGHIH-ROOHI S, HAJIZADEH S, et al. A Big Data Analysis Approach for Rail Failure Risk Assessment[J/OL]. Risk Analysis, 2017, 37(8): 1495-1507. DOI: 10.1111/ri sa.12836.
[38] VON SOLMS R,VANNIEKERK J. From information security to cyber security[J]. computers & security, 2013, 38: 97-102.
[39] SAMONAS S, COSS D. the Cia Strikes Back: Redefining Confidentiality, Integrity and Availability in Security[J/OL]. Journal of Information System Security, 2014, 10(3): 21-45. www.jissec.org.
[40] MOSER A, KRUEGEL C,KIRDAE. Exploring multiple execution paths for malware analysis [C]//2007 IEEE Symposium on Security and Privacy (SP’07). IEEE, 2007: 231-245.
[41] LYU M R, LAU L K. Firewall security: Policies, testing and performance evaluation [C]//Proceedings 24th Annual International Computer Software and Applications Conference. COMPSAC2000. IEEE, 2000: 116-121.
[42] KORET J, BACHAALANY E. The antivirus hacker’s handbook[M]. John Wiley & Sons, 2015.
[43] MORRIS T H, THORNTON Z, TURNIPSEED I P. Industrial Control System Simulation and Data Logging for Intrusion Detection System Research[C/OL]//2015. https://api.semanticscho lar.org/CorpusID:42986835.
[44] PONOMAREV S, ATKISON T. Industrial control system network intrusion detection by telemetry analysis[J]. IEEE Transactions on Dependable and Secure Computing, 2015, 13(2): 252-260.
[45] LIN C Y, NADJM-TEHRANI S, ASPLUND M. Timing-based anomaly detection in SCADA networks[C]//Critical Information Infrastructures Security: 12th International Conference, CRITIS 2017, Lucca, Italy, October 8-13, 2017, Revised Selected Papers 12. Springer, 2018: 48-59.
[46] CHEN Q, BRIDGES R A. Automated behavioral analysis of malware: A case study of wan- nacry ransomware[C]//2017 16th IEEE International Conference on machine learning and ap- plications (ICMLA). IEEE, 2017: 454-460.
[47] BONNER L. Cyber risk: How the 2011 Sony data breach and the need for cyber risk insurance policies should direct the federal response to rising data breaches[J]. Wash. UJL & Pol’y, 2012, 40: 257.
[48] MINKUS T, ROSS K W. I know what you ’re buying: Privacy breaches on ebay[C]//Privacy Enhancing Technologies: 14th International Symposium, PETS 2014, Amsterdam, The Nether- lands, July 16-18, 2014. Proceedings 14. Springer, 2014: 164-183.
[49] WU J. Cyberspace mimic defense[M]. Springer, 2020.
[50] FORREST S, BEAUCHEMIN C. Computer immunology[J]. Immunological reviews, 2007, 216(1): 176-197.
[51] ZHIWEN J, TAO L, AIQUN H. Research on Endogenous Security Methods of Embedded System[J/OL]. 2020 IEEE 6th International Conference on Computer and Communications, ICCC 2020, 2020: 1946-1950. DOI: 10.1109/ICCC51575.2020.9344972.
[52] XU M, GUO J, YUAN H, et al. Zero-Trust Security Authentication Based on SPA and Endoge- nous Security Architecture[J]. Electronics, 2023, 12(4): 782.
[53] GUO J, XU M. ZTESA—A Zero-Trust Endogenous Safety Architecture: Gain the endogenous safety benefit, avoid insider threats[C]//International Symposium on Computer Applications and Information Systems (ISCAIS 2022): Vol. 12250. SPIE, 2022: 192-202.
[54] YOU W, XU M, ZHOU D. Research on security protection technology for 5G cloud network [J/OL]. 2021 International Conference on Advanced Computing and Endogenous Security, ICACES 2021, 2021: 1-11. DOI: 10.1109/IEEECONF52377.2022.10013352.
[55] JI X, WU J, JIN L, et al. Discussion on a new paradigm of endogenous security towards 6G networks[J/OL]. Frontiers of Information Technology and Electronic Engineering, 2022, 23(10): 1421-1450. DOI: 10.1631/FITEE.2200060.
[56] MNIH V, KAVUKCUOGLU K, SILVER D, et al. Human-level control through deep rein- forcement learning[J]. nature, 2015, 518(7540): 529-533.
[57] LECUN Y, BENGIO Y, HINTON G. Deep learning[J]. nature, 2015, 521(7553): 436-444.
[58] SUTTON R S, BARTO A G. Reinforcement learning: An introduction[M]. MIT press, 2018.
[59] AN D, YANG Q, LIU W, et al. Defending against data integrity attacks in smart grid: A deep reinforcement learning-based approach[J]. IEEE Access, 2019, 7: 110835-110845.
[60] WEI F, WAN Z, HE H. Cyber-attack recovery strategy for smart grid based on deep reinforce- ment learning[J]. IEEE Transactions on Smart Grid, 2019, 11(3): 2476-2486.
[61] CAMINERO G, LOPEZ-MARTIN M, CARRO B. Adversarial environment reinforcement learning algorithm for intrusion detection[J]. Computer Networks, 2019, 159: 96-109.
[62] QIANG, LIU J. Development of deep reinforcement learning-based fault diagnosis method for rotating machinery in nuclear powerplants[J]. Progress in Nuclear Energy, 2022, 152: 104401.
[63] LIU Y, ZHANG D, GOOI H B. Optimization strategy based on deep reinforcement learning for home energy management[J]. CSEE Journal of Power and Energy Systems, 2020, 6(3): 572-582.
[64] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal Policy Optimization Algorithms [A/OL]. 2017: 1-12. arXiv: 1707.06347. http://arxiv.org/abs/1707.06347.
[65] ALMAHAMID F, GROLINGER K. Reinforcement learning algorithms: An overview and classification[C]//2021 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE). IEEE, 2021: 1-7.
[66] SCHULMAN J, MORITZ P, LEVINE S, et al. High-dimensional continuous control using generalized advantage estimation[A]. 2015.
[67] DOWNS J J, VOGEL E F. A plant-wide industrial process control problem[J/OL]. Computers and Chemical Engineering, 1993, 17(3): 245-255. DOI: 10.1016/0098-1354(93)80018-I.
[68] RICKER N L. Optimal steady-state operation of the Tennessee Eastman challenge process [J/OL]. Computers and Chemical Engineering, 1995, 19(9): 949-959. http://dx.doi.org/10.10 16/0098-1354(94)00043-N.
[69] RICKER N L. Tennessee Eastman Challenge Archive[EB/OL]. https://depts.washington.edu /control/LARRY/TE/download.html.
[70] ANDERSEN E B, UDUGAMA I A, GERNAEY K V, et al. An easy to use GUI for simu- lating big data using Tennessee Eastman process[J/OL]. Quality and Reliability Engineering International, 2022, 38(1): 264-282. DOI: 10.1002/qre.2975.
[71] BATHELT A, RICKER N L, JELALI M. Revision of the Tennessee eastman process model [J/OL]. IFAC-PapersOnLine, 2015, 28(8): 309-314. http://dx.doi.org/10.1016/j.ifacol.2015.0 8.199.
[72] REINARTZ C, ENEVOLDSEN T T. pyTEP: A Python package for interactive simulations of the Tennessee Eastman process[J/OL]. SoftwareX, 2022, 18: 101053. https://doi.org/10.1016/ j.softx.2022.101053.
[73] CHOU P W, MATURANA D, SCHERER S. Improving stochastic policy gradients in contin- uous control with deep reinforcement learning using the beta distribution[J]. 34th International Conference on Machine Learning, ICML 2017, 2017, 2: 1386-1396.
[74] PASZKEA, GROSS S, MASSA F, et al. Pytorch: An imperative style, high-performance deep learning library[J]. Advances in neural information processing systems, 2019, 32.
[75] GOLSHAN M, BOOZARJOMEHRY R B, PISHVAIE M R. A new approach to realtime op- timization of the Tennessee Eastman challenge problem[J/OL]. Chemical Engineering Journal, 2005, 112(1-3): 33-44. DOI: 10.1016/j.cej.2005.06.005.