FIDEDUPERANGE ioctl doesn't behave as expected on btrfs

Question

According to ioctl_fideduperange,

The maximum size of src_length is filesystem dependent and is typically 16 MiB.

However, I've been able use src_length of > 1 Gib successfully with a single call to ioctl.  Is that warning about 16 MiB just a complete exaggeration, at least for btrfs?

Also, according to the VFS documentation,

Implementations must handle callers passing in len == 0; this means “remap to the end of the source file”.

However, when I try setting src_length to 0, the ioctl call succeeds but without doing anything.

Am I misreading these two sentences, or does the btrfs implementation simply not conform (well) to the documentation? I'm testing on Linux Mint 20 with kernel 5.4.0-62-generic. I'm using filefrag -sv FILE1 FILE2 to check the block-level allocation of the files, to see if they're duplicated or not. I'm using the program below to deduplicate the files. The files in question are on a RAID-1 btrfs filesystem (on LUKS-encrypted partitions) created with sudo mkfs.btrfs -mraid1 -draid1 /dev/mapper/sda1_crypt /dev/mapper/sdb1_crypt.

Scenario:

$ cp -f file1 file2
$ filefrag -sv file1 file2   # see that files use different extents (are not deduplicated)
$ myprog file1 file2
$ filefrag -sv file1 file2   # see that files use the same extents (have been deduplicated)

Program to deduplicate two files:

// deduplicate srcfile and targetfile if contents are identical
// usage:  myprog srcfile targetfile
// compile with:  gcc myprog.c -o myprog

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <fcntl.h>
#include <sys/stat.h>
#include <sys/ioctl.h>
#include <linux/fs.h>

int main(int argc, char**argv)
{
   struct stat st;
   long size;
   __u64 buf[2048];        /* __u64 for proper field alignment */
   struct file_dedupe_range *range = (struct file_dedupe_range *)buf;

   memset(range, 0, sizeof(struct file_dedupe_range));
   memset(&range->info, 0, sizeof(struct file_dedupe_range_info));

   long srcfd = open(argv[1], O_RDONLY);
   if (srcfd < 0) { perror("open-src"); exit(1); }
   if (fstat(srcfd, &st) < 0) { perror("stat-src"); exit(1); }
   size = st.st_size;

   long tgtfd = open(argv[2], O_RDWR);
   if (tgtfd < 0) { perror("open-tgt"); exit(1); }
   if (fstat(tgtfd, &st) < 0) { perror("stat-tgt"); exit(1); }
   if (size != st.st_size) {
      fprintf(stderr, "SIZE DIFF\n");
      exit(1);
   }

   range->src_offset = 0;
   range->src_length = size;
// range->src_length = 0;                    // I expected this to work
   range->dest_count = 1;
   range->info[0].dest_fd = tgtfd;
   range->info[0].dest_offset = 0;

   while (range->src_length > 0) {
      if (ioctl(srcfd, FIDEDUPERANGE, range) < 0) { perror("ioctl"); exit(1); }

      fprintf(stderr, "bytes_deduped: %llu\n", range->info[0].bytes_deduped);
      fprintf(stderr, "status: %d\n", range->info[0].status);
      if (range->info[0].status == FILE_DEDUPE_RANGE_DIFFERS) {
         fprintf(stderr, "DIFFERS\n");
         break;
      } else if (range->info[0].status == FILE_DEDUPE_RANGE_SAME) {
         fprintf(stderr, "SAME\n");
      } else {
         fprintf(stderr, "ERROR\n");
         break;
      }

      if (range->info[0].bytes_deduped >= range->src_length) { break; }
      range->src_length -= range->info[0].bytes_deduped;
      range->src_offset += range->info[0].bytes_deduped;
      range->info[0].dest_offset += range->info[0].bytes_deduped;
   }
   exit(0);
}

Answer 1

Older versions of btrfs (e.g., 4.15) had a 16 MiB limit per FIDEDUPERANGE call, and would silently cut oversized requests down to 16 MiB. I forget exactly when the change happened, but the current version of btrfs (i.e., 5.16) loops in 16 MiB chunks. I think linux (not btrfs, now) still silently cuts down requests over 1 GiB, though. If you expect to use FIDEDUPERANGE with older versions of btrfs, you should definitely respect the 16 MiB limit. Also, other filesystems might have a similar limit.

As for src_length = 0, you should really consult the documentation for the individual ioctls for instructions on how to use them. The man page for FIDEDUPERANGE you quoted correctly documents that src_length = 0 means to dedup nothing.

Regarding the VFS page you quoted, things are just complicated. remap_file_range() handles the functionality for multiple ioctls adapted from btrfs that were originally designed and implemented separately in btrfs. In the clone ioctls, src_length == 0 means clone to the end of the file. In the dedup ioctl, src_length == 0 means dedup nothing. I forget exactly when, but there was an effort to unify the clone and dedup functions. However, it's not very nice to change the ioctl interface of currently supported versions of btrfs. In version 5.16, there's a weird hack involving btrfs_remap_file_range_prep() that converts the len argument to remap_file_range() depending on whether the ioctl was a clone or dedup call. It's tempting to say the VFS documentation is wrong, since I don't think the btrfs behavior here has changed. However, I'm not sure whether other filesystems have implemented remap_file_range() with this meaning of len == 0, so it's just complicated.