2

The size of "file" is showing 15TB when i check the size with ls -l

ls -l
total 16
-rw-r--r-- 1 root root 15393162788865 May 30 13:41 file

when i check the size of the "file" with du command, it shows the following.

du -a file 
12  file

After some googling, i came to a conclusion that the file may be a sparse file. commands like less, tail, cat, hexdump etc takes forever when i read it.

here is the output of filefrag.

filefrag -e file 
Filesystem type is: ef53
File size of file is 15393162788865 (3758096385 blocks of 4096 bytes)
 ext:       logical_offset:        physical_offset: length:   expected: flags:
   0:         0..        0:   22261760..  22261760:      1:            
   1: 3169274812..3169274812:   22268860..  22268860:      1: 3191536572:
   2: 3758096383..3758096383:   22271999..  22271999:      1:  611090431: last
file: 3 extents found

i want to know if there is a way to see only the contents of the file without the holes/zeros in it from Linux terminal.

  • 1
    @DarkHeart od doesn't know about sparse files. try dd if=/dev/null seek=30G of=/tmp/sparse; time od -x /tmp/sparse – mosvy May 30 at 13:54
0

On newer linux systems, there are the SEEK_DATA and SEEK_HOLE extension of lseek(2) which allow an app to skip "holes" when reading a sparse file.

On older systems, ioctl(FIBMAP) could be used and the data could be read directly from the underlying device (FIBMAP needs CAP_SYS_RAWIO capabilities, though).

Unfortunately, I'm not aware of any coreutils / standard utility making use of either.

Here is a small sparse_cat demo, which is using those in order to dump the data from very large sparse files in no time.

Example:

$ cc -Wall -O2 sparse_cat.c -s -o sparse_cat

$ echo 1st | dd conv=notrunc status=none bs=1 seek=10M of=/tmp/sparse
$ echo 2nd | dd conv=notrunc status=none bs=1 seek=10G of=/tmp/sparse
$ echo 3rd | dd conv=notrunc status=none bs=1 seek=10T of=/tmp/sparse
$ ls -l /tmp/sparse
-rw-r--r-- 1 ahq ahq 10995116277764 May 30 16:29 /tmp/sparse
$ ./sparse_cat </tmp/sparse >/dev/tty
          a00000           a01000
1st
       280000000        280001000
2nd
     a0000000000      a0000000004
3rd

Note: to keep things simple, I've omitted any file opening code (it should always be used as sparse_cat < input, not sparse_cat input) and any work-arounds for the bad interactions between sendfile(2) and ttys opened with the O_APPEND flag (use >/dev/tty explicitly).

Also notice that the data/hole ranges have block granularity -- the 1st string from the example above is actually followed by block size - 4 nul bytes.

sparse_cat.c

#define _GNU_SOURCE
#include <err.h>
#include <errno.h>
#include <fcntl.h>
#include <limits.h>
#include <stdio.h>
#include <unistd.h>
#include <sys/sendfile.h>

int main(void){
        off_t hole, data, pos, len; typedef unsigned long long ull;
        for(hole = 0;; data = hole){
                if((data = lseek(0, hole, SEEK_DATA)) == -1){
                        if(errno == ENXIO) return 0;
                        err(1, "lseek +data");
                }
                if((hole = lseek(0, data, SEEK_HOLE)) == -1)
                        err(1, "lseek +hole");
                dprintf(2, "%16llx %16llx\n", (ull)data, (ull)hole);
                for(pos = data; pos < hole;){
                        len = hole - pos; if(len > INT_MAX) len = INT_MAX;
                        if((len = sendfile(1, 0, &pos, len)) == 0) return 0;
                        if(len < 0) err(1, "sendfile");
                }
        }
}
  • Thanks @mosvy. It worked like a charm! – john May 30 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.