视频时域3D降噪算法与自适应帧缓存压缩技术的硬件架构设计

视频降噪是一项在多个领域中被广泛使用的重要技术，其目的是消除视频信 号中存在的噪声干扰。随着现代应用对高分辨率视频和实时处理的需求日益增长， 视频降噪算法的效率成为了实际应用中不可忽视的关键因素。在实践中，视频降 噪方法的速度和效果往往是业界和学术界关注的重点问题，需要不断优化算法设 计和实现方法，以提高视频降噪的实时性和降噪效果。

本文提出了一种高效的视频降噪算法，旨在利用自然视频序列中的时空冗余 性来提高降噪效果，该算法适用于多种图像格式，包括RAW 图像、灰度图像、RGB 和YUV 格式图像。本文提出的算法混合使用了2D 非线性滤波器和3D 时空滤波 器，在不同场景下均能表现出较好的降噪效果。具体来说，该算法首先对原始图像 进行中值滤波和双边滤波处理，以获取初步简单降噪的图像作为中间结果，用此 图像进行噪声估计。与此同时，为了对不同的运动区域实施不同的降噪策略，该 方案通过比较前一帧和当前帧来估计连续帧之间的运动状态。随后使用维纳滤波 结合噪声估计和运动检测信息，综合考虑两种信息对图像进行精细化的区域划分， 并且计算在不同的区域中前一帧和当前帧融合权重，从而实现当前帧的降噪处理。 这一步骤对单个像素进行处理，算法最终将加权融合后的像素返回到各自的位置， 得到降噪后的图像。该算法不仅能够实现高效的视频降噪处理，而且能够适用与 多种图像格式，具有一定的普适性和实用性。

此外，本文还针对该算法设计了一种像素级别流水线硬件架构，可用于实现 高质量、实时视频降噪的需求，具有成本低、功耗低、面积小等优势。该架构主要 分为两个部分：降噪模块和编解码存储模块。为了减少面积和功耗，我们通过引 入图像编解码器，降低了部分用于帧缓存的SRAM。为了更好地适应所提出的硬 件架构，本文还设计了一种自适应压缩质量的定制化图像压缩算法用于减小帧缓 存，可以有效地节省资源消耗。通过设置梯度排列的多个压缩质量阈值，该压缩 算法在不同的存储占用率情况下，能够动态的调整压缩质量，在保证SRAM 不会 溢出的情况下，得到最佳的图像质量。该架构已经在Stratix V 5SGXEA7H3F35C3 FPGA 平台上进行了功能验证，对于处理灰度1920 × 1080 分辨率视频的版本，其 硬件消耗如下：61688 个LUT、31632 个寄存器和4880kbit SRAM。

[1] XU J, ZHANG L, ZHANG D. External Prior Guided Internal Prior Learning for Real-WorldNoisy Image Denoising[J/OL]. IEEE Transactions on Image Processing, 2018, 27(6): 2996-3010. DOI: 10.1109/TIP.2018.2811546.
[2] ZHA Z, YUAN X, WEN B, et al. From Rank Estimation to Rank Approximation: Rank Residual Constraint for Image Restoration: arXiv:1807.02504[M]. arXiv, 2020.
[3] ZHANG L, ZUO W. Image Restoration: From Sparse and Low-Rank Priors to Deep Priors[Lecture Notes][J]. IEEE Signal Processing Magazine, 2017, 34(5): 172-179.
[4] TIRER T, GIRYES R. Back-Projection Based Fidelity Term for Ill-Posed Linear Inverse Problems[A/OL]. 2020. arXiv: 1906.06794.
[5] ZHANG K, ZUO W, CHEN Y, et al. Beyond a Gaussian Denoiser: Residual Learning of DeepCNN for Image Denoising[J/OL]. IEEE Transactions on Image Processing, 2017, 26(7): 3142-3155. DOI: 10.1109/TIP.2017.2662206.
[6] GUPTA D K, ARYA D, GAVVES E. Rotation Equivariant Siamese Networks for Tracking[C/OL]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville, TN, USA: IEEE, 2021: 12357-12366. DOI: 10.1109/CVPR46437.2021.01218.
[7] CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs:arXiv:1606.00915[M]. arXiv, 2017.
[8] RACZKOWSKA M K, KOZIOL P, Urbaniak-Wasik S, et al. Influence of Denoising on Classification Results in the Context of Hyperspectral Data: High Definition FT-IR Imaging[J/OL]. Analytica Chimica Acta, 2019, 1085: 39-47. DOI: 10.1016/j.aca.2019.07.045.
[9] BERNSTEIN R. Adaptive Nonlinear Filters for Simultaneous Removal of Different Kinds of Noise in Images[J]. IEEE Transactions on Circuits and Systems, 1987, 34(11): 1275-1291.
[10] XU J, ZHANG L, ZUO W, et al. Patch Group Based Nonlocal Self-Similarity Prior Learning for Image Denoising[C/OL]//2015 IEEE International Conference on Computer Vision (ICCV). Santiago, Chile: IEEE, 2015: 244-252. DOI: 10.1109/ICCV.2015.36.
[11] DE BONET J S. Noise Reduction through Detection of Signal Redundancy[J]. Rethinking Artificial Intelligence, MIT AI Lab, Tech. Rep, 1997.
[12] BORACCHI G, FOI A, et al. Multiframe Raw-Data Denoising Based on Block-Matching and 3-D Filtering for Low-Light Imaging and Stabilization[C]//Proc. Int. Workshop on Local and Non-Local Approx. in Image Processing: volume 1. Citeseer, 2008: 277-284.
[13] HUANG T. Stability of Two-Dimensional Recursive Filters[J]. IEEE Transactions on Audio and Electroacoustics, 1972, 20(2): 158-163.
[14] ZOHAIR A A, SHAMIL A A, SULONG G. Latest Methods of Image Enhancement andRestoration for Computed Tomography: A Concise Review[J]. Applied Medical Informatics,2015, 36(1): 1-12.
[15] LINDENBAUM M, FISCHER M, BRUCKSTEIN A. On Gabor’s Contribution to Image Enhancement[J]. Pattern recognition, 1994, 27(1): 1-8.
[16] LEVADA A L. Closed-Form Bayesian Image Denoising: Improving the Adaptive WienerFilter through Pairwise Gaussian-Markov Random Fields[J]. Communications in Statistics-Simulation and Computation, 2021, 50(4): 1094-1118.
[17] BENESTY J, CHEN J, HUANG Y. Study of the Widely Linear Wiener Filter for Noise Reduction[C/OL]//2010 IEEE International Conference on Acoustics, Speech and Signal Processing.Dallas, TX, USA: IEEE, 2010: 205-208. DOI: 10.1109/ICASSP.2010.5496033.
[18] PITAS I, VENETSANOPOULOS A. Nonlinear Mean Filters in Image Processing[J]. IEEEtransactions on acoustics, speech, and signal processing, 1986, 34(3): 573-584.
[19] HONG S W, BAO P. An Edge-Preserving Subband Coding Model Based on Non-Adaptive andAdaptive Regularization[J]. Image and Vision Computing, 2000, 18(8): 573-582.
[20] PITAS I, VENETSANOPOULOS A N. Nonlinear Digital Filters: Principles and Applications:volume 84[M]. Springer Science & Business Media, 2013.
[21] LU C T, CHOU T C. Denoising of Salt-and-Pepper Noise Corrupted Image Using ModifiedDirectional-Weighted-Median Filter[J]. Pattern Recognition Letters, 2012, 33(10): 1287-1295.
[22] TOMASI C, MANDUCHI R. Bilateral Filtering for Gray and Color Images[C/OL]//Sixth International Conference on Computer Vision (IEEE Cat. No.98CH36271). Bombay, India: Narosa Publishing House, 1998: 839-846. DOI: 10.1109/ICCV.1998.710815.
[23] BUADES A, COLL B, MOREL J M. A Review of Image Denoising Algorithms, with a New One[J/OL]. Multiscale Modeling & Simulation, 2005, 4(2): 490-530. DOI: 10.1137/040616024.
[24] CAI T T, SILVERMAN B W. Incorporating Information on Neighbouring Coefficients intoWavelet Estimation[J]. Sankhyā: The Indian Journal of Statistics, Series B, 2001: 127-148.
[25] MOHL B, WAHLBERG M, MADSEN P. Ideal Spatial Adaptation via Wavelet Shrinkage[J]. The Journal of the Acoustical Society of America, 2003, 114: 1143-1154.
[26] MALLAT S, HWANG W L. Singularity Detection and Processing with Wavelets[J]. IEEEtransactions on information theory, 1992, 38(2): 617-643.
[27] KAUR G, GARG M, GUPTA S, et al. Denoising of Images Using Thresholding Based onWavelet Transform Technique[C]//IOP Conference Series: Materials Science and Engineering:volume 1022. IOP Publishing, 2021: 012031.
[28] LEI S, LU M, LIN J, et al. Remote Sensing Image Denoising Based on Improved Semi-Soft Threshold[J]. Signal, Image And Video Processing, 2021, 15: 73-81.
[29] MAGGIONI M, BORACCHI G, FOI A, et al. Video Denoising, Deblocking, and Enhancement Through Separable 4-D Nonlocal Spatiotemporal Transforms[J/OL]. IEEE Transactions on Image Processing, 2012, 21(9): 3952-3966. DOI: 10.1109/TIP.2012.2199324.
[30] BUADES A, COLL B, MOREL J. Denoising Image Sequences Does Not Require Motion Estimation[C/OL]//Proceedings. IEEE Conference on Advanced Video and Signal Based Surveillance,2005. Como, Italy: IEEE, 2005: 70-74. DOI: 10.1109/AVSS.2005.1577245.
[31] DABOV K, FOI A, KATKOVNIK V, et al. Image Denoising by Sparse 3-D Transform-Domain Collaborative Filtering[J/OL]. IEEE Transactions on Image Processing, 2007, 16(8): 2080-2095. DOI: 10.1109/TIP.2007.901238.
[32] DABOV K, FOI A, EGIAZARIAN K. Video Denoising by Sparse 3D Transform-DomainCollaborative Filtering[C]//2007 15th European Signal Processing Conference. 2007: 145-149.
[33] WELCH G. An Introduction to the Kalman Filter[Z]. 1997: 16.
[34] PFLEGER S G, PLENTZ P D M, ROCHA R C O, et al. Real-Time Video Denoising on Multicores and GPUs with Kalman-Based and Bilateral Filters Fusion[J/OL]. Journal of Real-Time Image Processing, 2019, 16(5): 1629-1642. DOI: 10.1007/s11554-016-0659-y.
[35] EHMANN J, CHU L C, TSAI S F, et al. Real- Time Video Denoising on Mobile Phones[C/OL]//2018 25th IEEE International Conference on Image Processing (ICIP). Athens: IEEE, 2018:505-509. DOI: 10.1109/ICIP.2018.8451416.
[36] GU S, TIMOFTE R. A Brief Review of Image Denoising Algorithms and Beyond[J]. Inpainting and Denoising Challenges, 2019: 1-21.
[37] TASSANO M, DELON J, VEIT T. DVDnet: A Fast Network for Deep Video Denoising[C/OL]//2019 IEEE International Conference on Image Processing (ICIP). 2019: 1805-1809.DOI: 10.1109/ICIP.2019.8803136.
[38] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-Based Learning Applied to Document Recognition[J/OL]. Proceedings of the IEEE, Nov., 86(11): 2278-2324. DOI: 10.1109/5.726791.
[39] JAIN V, SEUNG S. Natural Image Denoising with Convolutional Networks[C]//Advances in Neural Information Processing Systems: volume 21. Curran Associates, Inc., 2008.
[40] BURGER H C, SCHULER C J, HARMELING S. Image Denoising: Can Plain Neural Networks Compete with BM3D?[C/OL]//2012 IEEE Conference on Computer Vision and PatternRecognition. Providence, RI: IEEE, 2012: 2392-2399. DOI: 10.1109/CVPR.2012.6247952.
[41] SCHMIDT U, ROTH S. Shrinkage Fields for Effective Image Restoration[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2014: 2774-2781.
[42] CHEN Y, POCK T. Trainable Nonlinear Reaction Diffusion: A Flexible Framework for Fast and Effective Image Restoration[J/OL]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017, 39(6): 1256-1272. DOI: 10.1109/TPAMI.2016.2596743.
[43] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet Classification with Deep Convolutional Neural Networks[J/OL]. Communications of the ACM, 2017, 60(6): 84-90. DOI:10.1145/3065386.
[44] MAO X, SHEN C, YANG Y B. Image Restoration Using Very Deep Convolutional Encoder-Decoder Networks with Symmetric Skip Connections[C]//Advances in Neural Information Processing Systems: volume 29. Curran Associates, Inc., 2016.
[45] ZIV J, LEMPEL A. Compression of Individual Sequences via Variable-Rate Coding[J]. IEEE transactions on Information Theory, 1978, 24(5): 530-536.
[46] STORER J A, SZYMANSKI T G. Data Compression via Textual Substitution[J/OL]. Journal of the ACM, 1982, 29(4): 928-951. DOI: 10.1145/322344.322346.
[47] FIALA E R, GREENE D H. Data Compression with Finite Windows[J/OL]. Communications of the ACM, 1989, 32(4): 490-505. DOI: 10.1145/63334.63341.
[48] WILLIAMS R N. An Extremely Fast Ziv-Lempel Data Compression Algorithm[C]//1991 Data Compression Conference. IEEE Computer Society, 1991: 362-363.
[49] WELCH T A. A Technique for High-Performance Data Compression[J]. Computer, 1984, 17(06): 8-19.
[50] HUFFMAN D A. A Method for the Construction of Minimum-Redundancy Codes[J]. Proceedings of the IRE, 1952, 40(9): 1098-1101.
[51] CHU J, CHEN Q, YANG X. Review on Full Reference Image Quality Assessment Algorithms[J]. Application Research of Computers, 2014, 31(1): 13-22.
[52] HORE A, ZIOU D. Image Quality Metrics: PSNR vs. SSIM[C/OL]//2010 20th International Conference on Pattern Recognition. Istanbul, Turkey: IEEE, 2010: 2366-2369. DOI: 10.1109/ICPR.2010.579.
[53] LIN S H, CHEN P Y, HSU C K. Modular Design of High-Efficiency Hardware Median Filter Architecture[J]. IEEE Transactions on Circuits and Systems I: Regular Papers, 2017, 65(6):1929-1940.
[54] CADENAS J O, MEGSON G M, SHERRATT R S. Median filter architecture by accumulative parallel counters[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2015, 62(7): 661-665.
[55] YAO R, CHEN L, DONG P, et al. A Compact Hardware Architecture for Bilateral Filter with the Combination of Approximate Computing and Look-up Table[J]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2022, 69(7): 3324-3328.
[56] HASINOFF S W, SHARLET D, GEISS R, et al. Burst Photography for High Dynamic Range and Low-Light Imaging on Mobile Cameras[J/OL]. ACM Transactions on Graphics, 2016, 35(6): 1-12. DOI: 10.1145/2980179.2980254.
[57] PEI Z, TONG Q, WANG L, et al. A Median Filter Method for Image Noise Variance Estimation[C/OL]//2010 Second International Conference on Information Technology and ComputerScience. 2010: 13-16. DOI: 10.1109/ITCS.2010.11.
[58] DONG P, CHEN Z, LI Z, et al. Configurable Image Rectification and Disparity Refinement for Stereo Vision[J/OL]. IEEE Transactions on Circuits and Systems II: Express Briefs, 2022, 69(10): 3973-3977. DOI: 10.1109/TCSII.2022.3191811.
[59] WILLÈME A, DESCAMPE A, LUGAN S, et al. Quality and Error Robustness Assessmentof Low-Latency Lightweight Intra-Frame Codecs for Screen Content Compression[J]. IEEEJournal on Emerging and Selected Topics in Circuits and Systems, 2016, 6(4): 471-483.
[60] ELBADRI M, PETERKIN R, GROZA V, et al. Hardware support of JPEG[C]//Canadian Conferenceon Electrical and Computer Engineering, 2005. IEEE, 2005: 812-815.
[61] SWAMINATHAN K, LAKSHMINARAYANAN G, KO S B. High speed generic network interface for network on chip using ping pong buffers[C]//2012 International Symposium on Electronic System Design (ISED). IEEE, 2012: 72-76.
[62] YUE H, CAO C, LIAO L, et al. Supervised raw video denoising with a benchmark dataset on dynamic scenes[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020: 2301-2310.