RUN: 基于 Rust 的用户空间高性能数据包处理框架

Rust 作为一种高级编程语言，能够在提供强大的内存安全保障同时，提供系统级编程能力。而 DPDK 和 XDP 等用户空间数据包 I/O 框架，能够提供较低的数据包延迟和大流量处理能力。针对网络流量迅速攀升以及网络安全威胁不断加剧等问题，科技公司、科研院所均投入了大量的研发资源，利用 Rust 编程语言以及用户空间数据包处理技术，来优化包括网络协议栈、 Web 服务器、网络功能软件在内的各类网络基础设施软件。但是，现有的优化大多在系统架构层面进行设计改良，而忽略了针对底层核心运行库的优化。一个典型的例子就是数据包处理库。用户空间数据平面通常需要线速处理上百 Gbps 的流量，在这一过程中，数据包处理库的总体运行效率将直接影响核心网络应用的性能。

在本文中，我们利用 Rust 编程语言，从零开始设计了一个全新的高性能数据包处理库 RUN。相比现有的 Rust 数据包处理库， RUN 提供了更加强大的数据包处理能力，即处理保存在非连续内存缓冲区上的数据包。同时， RUN 在实现上述能力的同时，可以保证内存安全性，并取得超越现有 Rust 数据包处理库的卓越性能。为了实现上述优势， RUN 在设计层面做出了三个主要贡献。第一， RUN 提出了一个名为数据包缓冲区的通用抽象，并定义了 7 个操作数据包缓冲区的方法。通过自由组合 7 个方法， RUN 可以实现复杂的数据包处理操作，并且实现非连续内存缓冲区的高效读写。利用该抽象， RUN 构建了一套通用的数据包处理 API，并且针对不同类型的内存缓冲区，定制了数据包缓冲区的具体实现。最终，不论数据包是否保存在非连续内存缓冲区上， RUN 都可以提供统一、高效的数据包处理能力。第二， RUN 利用数据包缓冲区充分隔离了 Rust 的不安全代码，并完全利用Rust 的安全代码实现了数据包处理 API。通过上述设计， RUN 取的了更好的内存安全性，可以有效抵抗由格式错误数据包引发的内存安全性问题。第三， RUN 基于隐式游标的理念，提出了两种相应的优化，显著提升了数据包缓冲区实现的性能。

我们通过详细的实验，验证了 RUN 的高效性。实验结果表明，使用 RUN 替换上层应用组件，可以为应用带来性能上的提升。具体表现为， TCP 协议栈的吞吐量最多提高了约 2Gbps 以及流量分析器的最大吞吐量提高了 3% 到 10% 左右。

[1] FOUNDATION L. Data Plane Development Kit[EB/OL]. 2018. https://www.dpdk.org/.

[2] HØILAND-JØRGENSEN T, BROUER J D, BORKMANN D, et al. The EXpress Data Path:Fast Programmable Packet Processing in the Operating System Kernel[C/OL]//CoNEXT ’18:Proceedings of the 14th International Conference on Emerging Networking EXperiments andTechnologies. New York, NY, USA: Association for Computing Machinery, 2018: 54– 66.https://doi.org/10.1145/3281411.3281443.

[3] PANDA A, HAN S, JANG K, et al. NetBricks: Taking the V out of NFV[C]//OSDI’16: Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation.USA: USENIX Association, 2016: 203–216.

[4] KAWASHIMA R, NAKAYAMA H, HAYASHI T, et al. Evaluation of Forwarding Effciencyin NFV-Nodes Toward Predictable Service Chain Performance[J/OL]. IEEE Transactions onNetwork and Service Management, 2017, 14(4): 920-933. DOI: 10.1109/TNSM.2017.2734560.

[5] QI S, ZENG Z, MONIS L, et al. MiddleNet: A Unified, High-Performance NFV and Middlebox Framework with eBPF and DPDK[J/OL]. IEEE Transactions on Network and ServiceManagement, 2023: 1-1. DOI: 10.1109/TNSM.2023.3256891.

[6] EMMERICH P, GALLENMüLLER S, RAUMER D, et al. MoonGen: A Scriptable High-SpeedPacket Generator[C/OL]//IMC ’15: Proceedings of the 2015 Internet Measurement Conference.New York, NY, USA: Association for Computing Machinery, 2015: 275–287. https://doi.org/10.1145/2815675.2815692.

[7] CHANG C H, LIN Y D, LAI Y K, et al. A scalable and accurate distributed traffc generatorwith Fourier transformed distribution over multiple commodity platforms[J/OL]. Journal ofNetwork and Computer Applications, 2019, 144: 102-117. https://www.sciencedirect.com/science/article/pii/S1084804519302255. DOI: https://doi.org/10.1016/j.jnca.2019.07.001.

[8] ZHANG T, LINGUAGLOSSA L, GALLO M, et al. FlowMon-DPDK: Parsimonious Per-FlowSoftware Monitoring at Line Rate[C/OL]//2018 Network Traffc Measurement and AnalysisConference (TMA). 2018: 1-8. DOI: 10.23919/TMA.2018.8506565.

[9] LINDNER S, HäBERLE M, MENTH M. P4TG: 1 Tb/s Traffc Generation for Ethernet/IPNetworks[J/OL]. IEEE Access, 2023, 11: 17525-17535. DOI: 10.1109/ACCESS.2023.3246262.

[10] WANG X, XU C, JIN W, et al. A Scalable Parallel Architecture Based on Many-Core Processorsfor Generating HTTP Traffc[J/OL]. Applied Sciences, 2017, 7(2). https://www.mdpi.com/2076-3417/7/2/154. DOI: 10.3390/app7020154.

[11] KONSTANTYNOWICZ M. FD. io-How to Push Extreme Limits of Performanceand Scale with Vector Packet Processing Technology[J]. url: https://www. ietf.org/proceedings/96/slides/slides-96-bmwg-10. pdf, 2020.46参考文献

[12] ZHENG C, TANG Q, LU Q, et al. Janus: A User-Level TCP Stack for Processing 40 MillionConcurrent TCP Connections[C/OL]//2018 IEEE International Conference on Communications(ICC). 2018: 1-7. DOI: 10.1109/ICC.2018.8422993.

[13] JEONG E, WOOD S, JAMSHED M, et al. mTCP: a Highly Scalable User-level TCP Stackfor Multicore Systems[C/OL]//11th USENIX Symposium on Networked Systems Design andImplementation (NSDI 14). Seattle, WA: USENIX Association, 2014: 489-502. https://www.usenix.org/conference/nsdi14/technical-sessions/presentation/jeong.

[14] KLABNIK S, NICHOLS C. The Rust programming language[M]. No Starch Press, 2023.

[15] KULKARNI C, MOORE S, NAQVI M, et al. Splinter: Bare-Metal Extensions for MultiTenant Low-Latency Storage[C/OL]//13th USENIX Symposium on Operating Systems Design and Implementation (OSDI 18). Carlsbad, CA: USENIX Association, 2018: 627-643.https://www.usenix.org/conference/osdi18/presentation/kulkarni.

[16] KöSTER J. Rust-Bio: a fast and safe bioinformatics library[J/OL]. Bioinformatics, 2015, 32(3): 444-446. https://doi.org/10.1093/bioinformatics/btv573.

[17] NARAYANAN V, HUANG T, DETWEILER D, et al. RedLeaf: Isolation and Communicationin a Safe Operating System[C/OL]//14th USENIX Symposium on Operating Systems Designand Implementation (OSDI 20). USENIX Association, 2020: 21-39. https://www.usenix.org/conference/osdi20/presentation/narayanan-vikram.

[18] BOOS K, ZHONG L. Theseus: A State Spill-Free Operating System[C/OL]//PLOS’17: Proceedings of the 9th Workshop on Programming Languages and Operating Systems. New York,NY, USA: Association for Computing Machinery, 2017: 29–35. https://doi.org/10.1145/3144555.3144560.

[19] LEVY A, CAMPBELL B, GHENA B, et al. Multiprogramming a 64kB Computer Safelyand Effciently[C/OL]//SOSP ’17: Proceedings of the 26th Symposium on Operating SystemsPrinciples. New York, NY, USA: Association for Computing Machinery, 2017: 234– 251.https://doi.org/10.1145/3132747.3132786.

[20] CONTRIBUTORS R O. Redox OS[EB/OL]. Accessed 2023. https://github.com/redox-os/redox.

[21] ANDERSON B, BERGSTROM L, GOREGAOKAR M, et al. Engineering the Servo WebBrowser Engine Using Rust[C/OL]//ICSE ’16: Proceedings of the 38th International Conference on Software Engineering Companion. New York, NY, USA: Association for ComputingMachinery, 2016: 81–89. https://doi.org/10.1145/2889160.2889229.

[22] NARAYAN S, DISSELKOEN C, GARFINKEL T, et al. Retrofitting Fine Grain Isolation in theFirefox Renderer[C/OL]//29th USENIX Security Symposium (USENIX Security 20). USENIXAssociation, 2020: 699-716. https://www.usenix.org/conference/usenixsecurity20/presentation/narayan.

[23] PIZENBERG M. Interactive computer vision through the Web[D]. 2020.

[24] BOUCHER S, KALIA A, ANDERSEN D G, et al. Putting the ”Micro” Back in Microservice[C/OL]//2018 USENIX Annual Technical Conference (USENIX ATC 18). Boston, MA:USENIX Association, 2018: 645-650. https://www.usenix.org/conference/atc18/presentation/boucher.

[25] VAHLDIEK-OBERWAGNER A, VIJ M. Meshwa: The case for a memory-safe software andhardware architecture for serverless computing[A]. 2022.

[26] HARCHOL Y, MUSHTAQ A, FANG V, et al. Making Edge-Computing Resilient[C/OL]//SoCC ’20: Proceedings of the 11th ACM Symposium on Cloud Computing. New York, NY,USA: Association for Computing Machinery, 2020: 253–266. https://doi.org/10.1145/3419111.3421278.

[27] RIZZO L. netmap: A Novel Framework for Fast Packet I/O[C/OL]//2012 USENIX AnnualTechnical Conference (USENIX ATC 12). Boston, MA: USENIX Association, 2012: 101-112.https://www.usenix.org/conference/atc12/technical-sessions/presentation/rizzo.

[28] CENTER M S R. 2019 MSRC Security Research Report[EB/OL]. 2019. https://www.microsoft.com/security/blog/2019/11/26/2019-msrc-security-research-report-now-available/.

[29] OREBAUGH A, RAMIREZ G, BEALE J. Wireshark & Ethereal network protocol analyzertoolkit[M]. Elsevier, 2006.

[30] CORPORATION T M. CVE-2019-12255: libpcap Memory Leak Vulnerability[EB/OL]. 2019.https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-12255.

[31] NIST. The Linux Kernels security flaw[M/OL]. National Vulnerability Database, 2016. https://nvd.nist.gov/vuln/detail/CVE-2016-8655.

[32] NIST. GNU C Library (glibc) getaddrinfo() stack-based buffer overflow[M/OL]. National Vulnerability Database, 2018. https://nvd.nist.gov/vuln/detail/CVE-2018-1000115.

[33] NIST. OpenBSD firewall security flaw[M/OL]. National Vulnerability Database, 2019. https://nvd.nist.gov/vuln/detail/CVE-2019-12456.

[34] JUNG R, TOLNAY D, JOURDAN J H, et al. Miri: A Rust interpreter with a focus on safetyand security[EB/OL]. 2021. https://github.com/rust-lang/miri.

[35] VANHATTUM A, SCHWARTZ-NARBONNE D, CHONG N, et al. Verifying Dynamic TraitObjects in Rust[C/OL]//ICSE-SEIP ’22: Proceedings of the 44th International Conference onSoftware Engineering: Software Engineering in Practice. New York, NY, USA: Association forComputing Machinery, 2022: 321–330. https://doi.org/10.1145/3510457.3513031.

[36] RIVERA E, MERGENDAHL S, SHROBE H, et al. Keeping Safe Rust Safe with Galeed[C/OL]//ACSAC ’21: Annual Computer Security Applications Conference. New York, NY,USA: Association for Computing Machinery, 2021: 824–836. https://doi.org/10.1145/3485832.3485903.

[37] JUNG R, JOURDAN J H, KREBBERS R, et al. RustBelt: Securing the Foundations of the RustProgramming Language[J/OL]. Proc. ACM Program. Lang., 2017, 2(POPL). https://doi.org/10.1145/3158154.

[38] DANG H H, JOURDAN J H, KAISER J O, et al. RustBelt Meets Relaxed Memory[J/OL]. Proc.ACM Program. Lang., 2019, 4(POPL). https://doi.org/10.1145/3371102.

[39] ASTRAUSKAS V, MüLLER P, POLI F, et al. Leveraging Rust Types for Modular Specificationand Verification[J/OL]. Proc. ACM Program. Lang., 2019, 3(OOPSLA). https://doi.org/10.1145/3360573.48参考文献

[40] TOMAN J, PERNSTEINER S, TORLAK E. Crust: A Bounded Verifier for Rust (N)[C/OL]//2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE).2015: 75-80. DOI: 10.1109/ASE.2015.77.

[41] ASTRAUSKAS V, MATHEJA C, POLI F, et al. How Do Programmers Use Unsafe Rust?[J/OL].Proc. ACM Program. Lang., 2020, 4(OOPSLA). https://doi.org/10.1145/3428204.

[42] WANG H, WANG P, DING Y, et al. Towards Memory Safe Enclave Programming with RustSGX[C/OL]//CCS ’19: Proceedings of the 2019 ACM SIGSAC Conference on Computer andCommunications Security. New York, NY, USA: Association for Computing Machinery, 2019:2333–2350. https://doi.org/10.1145/3319535.3354241.

[43] EMRE M. Translating C to Safe Rust: Reasoning about Pointer Types and Lifetimes[D]. UCSanta Barbara, 2022.

[44] DEWEY K, ROESCH J, HARDEKOPF B. Fuzzing the Rust Typechecker Using CLP (T)[C/OL]//2015 30th IEEE/ACM International Conference on Automated Software Engineering(ASE). 2015: 482-493. DOI: 10.1109/ASE.2015.65.

[45] LIU P, ZHAO G, HUANG J. Securing Unsafe Rust Programs with XRust[C/OL]//ICSE ’20:Proceedings of the ACM/IEEE 42nd International Conference on Software Engineering. NewYork, NY, USA: Association for Computing Machinery, 2020: 234–245. https://doi.org/10.1145/3377811.3380325.

[46] CUI M, CHEN C, XU H, et al. SafeDrop: Detecting Memory Deallocation Bugs of RustPrograms via Static Data-Flow Analysis[J/OL]. ACM Trans. Softw. Eng. Methodol., 2022.https://doi.org/10.1145/3542948.

[47] BAE Y, KIM Y, ASKAR A, et al. Rudra: Finding Memory Safety Bugs in Rust at the EcosystemScale[C/OL]//SOSP ’21: Proceedings of the ACM SIGOPS 28th Symposium on OperatingSystems Principles. New York, NY, USA: Association for Computing Machinery, 2021: 84–99. https://doi.org/10.1145/3477132.3483570.

[48] LI Z, WANG J, SUN M, et al. MirChecker: Detecting Bugs in Rust Programs via Static Analysis[C/OL]//CCS ’21: Proceedings of the 2021 ACM SIGSAC Conference on Computer and Communications Security. New York, NY, USA: Association for Computing Machinery, 2021:2183–2196. https://doi.org/10.1145/3460120.3484541.

[49] TEAM C D. Capsule: a Rust library for packet processing and network interface control[EB/OL]. 2021. https://github.com/capsule-rs/capsule.

[50] SCHMID C. etherparse: a packet sniffng and packet parsing library[EB/OL]. 2021. https://crates.io/crates/etherparse.

[51] DEVELOPERS P. Pnet: A Rust library for low-level networking[EB/OL]. 2018. https://github.com/libpnet/libpnet.

[52] LABS CONTRIBUTORS M. Smoltcp: A small TCP/IP stack implementation for embeddedsystems[EB/OL]. 2021. https://github.com/m-labs/smoltcp.

[53] WAN G, GONG F, BARBETTE T, et al. Retina: Analyzing 100GbE Traffc on Commodity Hardware[C/OL]//SIGCOMM ’22: Proceedings of the ACM SIGCOMM 2022 Conference. New York, NY, USA: Association for Computing Machinery, 2022: 530– 544. https://doi.org/10.1145/3544216.3544227.

[54] FLOYD S. HighSpeed TCP for large congestion windows[R/OL]. 2003. https://www.rfc-editor.org/rfc/rfc3649.html.

[55] LEITH D, SHORTEN R. H-TCP: TCP for high-speed and long-distance networks[Z]. 2004.

[56] HA S, RHEE I, XU L. CUBIC: A New TCP-Friendly High-Speed TCP Variant[J/OL]. SIGOPSOper. Syst. Rev., 2008, 42(5): 64–74. https://doi.org/10.1145/1400097.1400105.

[57] MOON Y, LEE S, JAMSHED M A, et al. AccelTCP: Accelerating Network Applications withStateful TCP Offloading[C/OL]//17th USENIX Symposium on Networked Systems Design andImplementation (NSDI 20). Santa Clara, CA: USENIX Association, 2020: 77-92. https://www.usenix.org/conference/nsdi20/presentation/moon.

[58] ZHANG H, ZHANG H, ZHANG L, et al. FastUDP: a highly scalable user-level UDP frameworkin multi-core systems for fast packet I/O[J/OL]. The Journal of Supercomputing, 2021, 77(5):5148-5175. https://doi.org/10.1007/s11227-020-03486-6.

[59] WU H, FENG Z, GUO C, et al. ICTCP: Incast Congestion Control for TCP in Data CenterNetworks[C/OL]//Co-NEXT ’10: Proceedings of the 6th International COnference. New York,NY, USA: Association for Computing Machinery, 2010. https://doi.org/10.1145/1921168.1921186.

[60] LÜKE K. Memory-safe Network Services Through A Userspace Networking Switch[D].KAIST, 2019.

[61] ZOU Y H, BAI J J, ZHOU J, et al. TCP-Fuzz: Detecting Memory and Semantic Bugs in TCPStacks with Fuzzing[C/OL]//2021 USENIX Annual Technical Conference (USENIX ATC 21).USENIX Association, 2021: 489-502. https://www.usenix.org/conference/atc21/presentation/zou.

[62] IPERF DEVELOPMENT TEAM T. iPerf2[EB/OL]. 2003. https://sourceforge.net/projects/iperf2/.

[63] WILES K. The DPDK Pktgen Application - Documentation[EB/OL]. 2019. https://pktgen-dpdk.readthedocs.io/en/latest/.

[64] COVINGTON G A, GIBB G, LOCKWOOD J W, et al. A Packet Generator on the NetFPGAPlatform[C/OL]//2009 17th IEEE Symposium on Field Programmable Custom Computing Machines. 2009: 235-238. DOI: 10.1109/FCCM.2009.29.

[65] PAXSON V. Bro: a system for detecting network intruders in real-time[J/OL]. ComputerNetworks, 1999, 31(23): 2435-2463. https://www.sciencedirect.com/science/article/pii/S1389128699001127. DOI: https://doi.org/10.1016/S1389-1286(99)00112-7.

[66] ROESCH M. Snort - Lightweight Intrusion Detection for Networks[C]//LISA ’99: Proceedingsof the 13th USENIX Conference on System Administration. USA: USENIX Association, 1999:229–238.

[67] JUNG R, JOURDAN J H, KREBBERS R, et al. Safe Systems Programming in Rust: ThePromise and the Challenge[J/OL]. Communications of the ACM, 2021, 64(4): 144-152. https://hal.science/hal-03021536. DOI: 10.1145/3418295.

[68] BROOK A, KRYUCHKOV V. Criterion.rs: Statistics-driven Rust micro-benchmarking[EB/OL]. 2021. https://github.com/bheisler/criterion.rs.

[69] MOLNáR S, MEGYESI P, SZABó G. How to validate traffc generators?[C/OL]//2013 IEEEInternational Conference on Communications Workshops (ICC). 2013: 1340-1344. DOI: 10.1109/ICCW.2013.6649445.