4

The size of "file" is showing 15TB when i check the size with ls -l

ls -l
total 16
-rw-r--r-- 1 root root 15393162788865 May 30 13:41 file

when i check the size of the "file" with du command, it shows the following.

du -a file 
12  file

After some googling, i came to a conclusion that the file may be a sparse file. commands like less, tail, cat, hexdump etc takes forever when i read it.

here is the output of filefrag.

filefrag -e file 
Filesystem type is: ef53
File size of file is 15393162788865 (3758096385 blocks of 4096 bytes)
 ext:       logical_offset:        physical_offset: length:   expected: flags:
   0:         0..        0:   22261760..  22261760:      1:            
   1: 3169274812..3169274812:   22268860..  22268860:      1: 3191536572:
   2: 3758096383..3758096383:   22271999..  22271999:      1:  611090431: last
file: 3 extents found

i want to know if there is a way to see only the contents of the file without the holes/zeros in it from Linux terminal.

1
  • 1
    @DarkHeart od doesn't know about sparse files. try dd if=/dev/null seek=30G of=/tmp/sparse; time od -x /tmp/sparse
    – user313992
    Commented May 30, 2019 at 13:54

1 Answer 1

3

On newer linux systems, there are the SEEK_DATA and SEEK_HOLE extension of lseek(2) which allow an app to skip "holes" when reading a sparse file.

On older systems, ioctl(FIBMAP) could be used and the data could be read directly from the underlying device (FIBMAP needs CAP_SYS_RAWIO capabilities, though).

Unfortunately, I'm not aware of any coreutils / standard utility making use of either.

Here is a small sparse_cat demo, which is using those in order to dump the data from very large sparse files in no time.

Example:

$ cc -Wall -O2 sparse_cat.c -s -o sparse_cat

$ echo 1st | dd conv=notrunc status=none bs=1 seek=10M of=/tmp/sparse
$ echo 2nd | dd conv=notrunc status=none bs=1 seek=10G of=/tmp/sparse
$ echo 3rd | dd conv=notrunc status=none bs=1 seek=10T of=/tmp/sparse
$ ls -l /tmp/sparse
-rw-r--r-- 1 ahq ahq 10995116277764 May 30 16:29 /tmp/sparse
$ ./sparse_cat </tmp/sparse >/dev/tty
          a00000           a01000
1st
       280000000        280001000
2nd
     a0000000000      a0000000004
3rd

Note: to keep things simple, I've omitted any file opening code (it should always be used as sparse_cat < input, not sparse_cat input) and any work-arounds for the bad interactions between sendfile(2) and ttys opened with the O_APPEND flag (use >/dev/tty explicitly).

Also notice that the data/hole ranges have block granularity -- the 1st string from the example above is actually followed by block size - 4 nul bytes.

sparse_cat.c

#define _GNU_SOURCE
#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/sendfile.h>

int main(void){
        off_t hole, data, pos, len; typedef unsigned long long ull;
        for(hole = 0;; data = hole){
                if((data = lseek(0, hole, SEEK_DATA)) == -1){
                        if(errno == ENXIO) return 0;
                        err(1, "lseek +data");
                }
                if((hole = lseek(0, data, SEEK_HOLE)) == -1)
                        err(1, "lseek +hole");
                dprintf(2, "%16llx %16llx\n", (ull)data, (ull)hole);
                for(pos = data; pos < hole;){
                        len = hole - pos; if(len > INT_MAX) len = INT_MAX;
                        if((len = sendfile(1, 0, &pos, len)) == 0) return 0;
                        if(len < 0) err(1, "sendfile");
                }
        }
}
1
  • Thanks @mosvy. It worked like a charm!
    – Akbar
    Commented May 30, 2019 at 14:14

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .