The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems

Deng，Cai1; Zou，Xiangyu1; Chen，Qi1; Tang，Bo2; Xia，Wen1

2024

10.1109/TC.2024.3363456

IEEE Transactions on Computers   影响因子和分区

0018-9340

1557-9956

Image data storage has grown explosively, so image deduplication is used to save storage by eliminating redundancy between different images. However, traditional image deduplication cannot eliminate fine-grained redundancy nor guarantee lossless results. In this work, we propose imDedup, a lossless and fine-grained deduplication scheme for JPEG image storage systems. Specifically, imDedup uses a novel sampling hash method, Feature Bitmap, to detect similar images in a fast way by utilizing the information distribution of JPEG data. Meanwhile, it uses Idelta, a novel delta encoder that incorporates image compression into deduplication, to guarantee the non-redundant data can be re-compressed via image encoding and thus improves the compression ratio. Besides, we propose the DCHash and Fixed-Point Matching (FPM) techniques to further speed up Idelta. We also propose imDedup-plus, which dynamically chooses the DCHash-based or FPM-based compressor to achieve higher throughputs without sacrificing the compression ratio. Experimental results demonstrate the superiority of the imDedup-based methods on five datasets. Compared with the state-of-the-art similarity detector and delta encoder, imDedup achieves 1.8–4.4× higher throughputs and 1.3–1.7× higher compression ratios, respectively. Besides, imDedup-plus can further achieve 1.3–2.9× higher throughputs than imDedup without sacrificing the compression ratio.

delta compression Discrete cosine transforms Feature extraction fine-grained deduplication Image coding image deduplication JPEG compression Quantization (signal) Redundancy storage systems Throughput Transform coding

SCI ; EI

英语

其他

COMPUTER SCIENCE

2-s2.0-85184826650

Scopus

1.Harbin Institute of Technology, Shenzhen, Guangdong, China
2.Southern University of Science and Technology, Guangdong, China

Deng，Cai,Zou，Xiangyu,Chen，Qi,et al. The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems[J]. IEEE Transactions on Computers,2024,73(5):1385-1399.

Deng，Cai,Zou，Xiangyu,Chen，Qi,Tang，Bo,&Xia，Wen.(2024).The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems.IEEE Transactions on Computers,73(5),1385-1399.

Deng，Cai,et al."The Design of a Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy for JPEG Image Storage Systems".IEEE Transactions on Computers 73.5(2024):1385-1399.