imDedup: A Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy among Images

Deng，Cai1; Chen，Qi1; Zou，Xiangyu1; Xu，Erci2; Tang，Bo3; Xia，Wen1

10.1109/ICDE53745.2022.00085

2022

38th IEEE International Conference on Data Engineering (ICDE)

1084-4627

978-1-6654-0884-4

Proceedings - International Conference on Data Engineering

2022-May

1071-1084

9-12 May 2022

Kuala Lumpur, Malaysia

10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA

IEEE COMPUTER SOC

Images occupy a large amount of storage in data centers. To cope with the explosive growth of the image storage requirement, image compression techniques are devised to shrink the size of every single image at first. Furthermore, image deduplication methods are proposed to reduce the storage cost as they could be used to eliminate redundancy among images. However, state-of-the-art image deduplication methods either can only eliminate file-level coarse-grained redundancy or cannot guarantee lossless deduplication. In this work, we propose a new lossless image deduplication framework to eliminate fine-grained redundancy among images. It first decodes images to expose similarity, then eliminates fine-grained redundancy on the decoded data by delta compres-sion, and finally re-compresses the remaining data by image compression encoding. Based on this framework, we propose a novel lossless similarity-based deduplication (SBD) scheme for decoded image data (called imDedup). Specifically, imDedup uses a novel and fast sampling method (called Feature Map) to detect similar images in a two-dimensional way, which greatly reduces computation overhead. Meanwhile, it uses a novel delta encoder (called Idelta) which incorporates image compression encoding characteristics into deduplication to guarantee the remaining deduplicated image data to be friendly re-compressed via image encoding, which significantly improves the compression ratio. We implement a prototype of imDedup for JPEG images, and demonstrate its superiority on four datasets: Compared with exact image deduplication, imDedup achieves a 19%-38% higher compression ratio by efficiently eliminating fine-grained redundancy. Compared with the similarity detector and delta encoder of state-of-the-art SBD schemes running on the decoded image data, imDedup achieves a 1.8×-3.4× higher throughput and a 1.3 ×-1. 6 × higher compression ratio, respectively.

fine grained deduplication image deduplication similarity based deduplication

其他

英语

CPCI-S ; EI

NSFC[61972441]

Computer Science

Computer Science, Artificial Intelligence ; Computer Science, Information Systems ; Computer Science, Theory & Methods

WOS:000855078401011

20223512637902

Cost reduction ; Decoding ; Digital storage ; Encoding (symbols) ; Image coding ; Image compression ; Image enhancement ; Signal encoding

Information Theory and Signal Processing:716.1 ; Data Storage, Equipment and Techniques:722.1 ; Data Processing and Image Processing:723.2

2-s2.0-85136441662

Scopus

1.Harbin Institute of Technology,Shenzhen,China
2.National University of Defense Technology,China
3.Southern University of Science and Technology,China

Deng，Cai,Chen，Qi,Zou，Xiangyu,et al. imDedup: A Lossless Deduplication Scheme to Eliminate Fine-grained Redundancy among Images[C]. 10662 LOS VAQUEROS CIRCLE, PO BOX 3014, LOS ALAMITOS, CA 90720-1264 USA:IEEE COMPUTER SOC,2022:1071-1084.