2

According to ioctl_fideduperange,

The maximum size of src_length is filesystem dependent and is typically 16 MiB.

However, I've been able use src_length of > 1 Gib successfully with a single call to ioctl. Is that warning about 16 MiB just a complete exaggeration, at least for btrfs?

Also, according to the vfs documentation,

Implementations must handle callers passing in len == 0; this means “remap to the end of the source file”.

However, when I try setting src_length to 0, the ioctl call succeeds but without doing anything.

Am I misreading these two sentences, or does the btrfs implementation simply not conform (well) to the documentation? I'm testing on Linux Mint 20 with kernel 5.4.0-62-generic. I'm using filefrag -sv FILE1 FILE2 to check the block-level allocation of the files, to see if they're duplicated or not. I'm using the program below to deduplicate the files. The files in question are on a RAID-1 btrfs filesystem (on LUKS-encrypted partitions) created with sudo mkfs.btrfs -mraid1 -draid1 /dev/mapper/sda1_crypt /dev/mapper/sdb1_crypt.

Scenario:

$ cp -f file1 file2
$ filefrag -sv file1 file2   # see that files use different extents (are not deduplicated)
$ myprog file1 file2
$ filefrag -sv file1 file2   # see that files use the same extents (have been deduplicated)

Program to deduplicate two files:

// deduplicate srcfile and targetfile if contents are identical
// usage:  myprog srcfile targetfile
// compile with:  gcc myprog.c -o myprog

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <linux/fs.h>

int main(int argc, char**argv)
{
   struct stat st;
   long size;
   __u64 buf[2048];        /* __u64 for proper field alignment */
   struct file_dedupe_range *range = (struct file_dedupe_range *)buf;

   memset(range, 0, sizeof(struct file_dedupe_range));
   memset(&range->info, 0, sizeof(struct file_dedupe_range_info));

   long srcfd = open(argv[1], O_RDONLY);
   if (srcfd < 0) { perror("open-src"); exit(1); }
   if (fstat(srcfd, &st) < 0) { perror("stat-src"); exit(1); }
   size = st.st_size;

   long tgtfd = open(argv[2], O_RDWR);
   if (tgtfd < 0) { perror("open-tgt"); exit(1); }
   if (fstat(tgtfd, &st) < 0) { perror("stat-tgt"); exit(1); }
   if (size != st.st_size) {
      fprintf(stderr, "SIZE DIFF\n");
      exit(1);
   }

   range->src_offset = 0;
   range->src_length = size;
// range->src_length = 0;                    // I expected this to work
   range->dest_count = 1;
   range->info[0].dest_fd = tgtfd;
   range->info[0].dest_offset = 0;

   while (range->src_length > 0) {
      if (ioctl(srcfd, FIDEDUPERANGE, range) < 0) { perror("ioctl"); exit(1); }

      fprintf(stderr, "bytes_deduped: %llu\n", range->info[0].bytes_deduped);
      fprintf(stderr, "status: %d\n", range->info[0].status);
      if (range->info[0].status == FILE_DEDUPE_RANGE_DIFFERS) {
         fprintf(stderr, "DIFFERS\n");
         break;
      } else if (range->info[0].status == FILE_DEDUPE_RANGE_SAME) {
         fprintf(stderr, "SAME\n");
      } else {
         fprintf(stderr, "ERROR\n");
         break;
      }

      if (range->info[0].bytes_deduped >= range->src_length) { break; }
      range->src_length -= range->info[0].bytes_deduped;
      range->src_offset += range->info[0].bytes_deduped;
      range->info[0].dest_offset += range->info[0].bytes_deduped;
   }
   exit(0);
}
2
  • After experimenting on a 30Gb file, I can see that the implementation of FIDEDUPERANGE has horrible performance. Not only does it have to verify that the files have identical contents, but it also has to reorganize the extent structure of the target file to match the source file. Neither is it atomic over the entire range of the file, just (I presume) on an extent by extent basis.
    – jrw32982
    Jan 19 at 5:00
  • FICLONE is much better performance-wise, but you, not the kernel, are responsible for making sure the data matches. Also FICLONE has the unwanted side effect of changing the target file's timestamps, which can be fixed with fstat + futimens.
    – jrw32982
    Jan 19 at 5:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.