1

Is there a command that allows me to "see" the raw bits of a disk? In other words, to ignore the filesystem and output the "big picture" of the literal zeros and ones as per the disk. (Ideally not requiring a TB-size temporary storage.)

What I'm actually looking for is a way to easily tell if sections of the disk contains long-repeating simple sequences, like all 0s, all 1s, all 01s, all 0101011s, etc, etc.

What's the best way to go about doing this?

  • What level of "raw" do you mean? You won't be able to get at the actual magnetic flux stored on the disk, the best you can do (with dd etc., as outlined below) is to get a complete block (which is already processed). Looking for long repeating simple sequences in such a block will probably not make sense. Is this an XY problem? Why do you need to look for repeating simple sequences? – dirkt Mar 5 '18 at 12:18
  • @dirkt, As raw as forensically possible. What do you mean by "already processed"? ¶ Re "why"; to research the differences in state between new disks, amongst others. – Pacerier Mar 9 '18 at 16:22
  • If you want "as raw as forensically possible", you need to take out the platters in a clean room and read them directly. Modern harddisks are heavily encoded, and the contents the harddisk controller sends to the host computer will only have a passing resemblance to what is on the platter. With that, the high capacities hard disks have today wouldn't be possible. If you want to research differences in state between new disks, forget reading contents. The best you can do is ask the controller, via SMART, or via vendor specific commands (which the vendor won't tell you). – dirkt Mar 9 '18 at 17:59
  • @dirkt, Do you mean that SMART uses vendor-specific commands? Surely there got to be some "industry standard" which is shared amongst different disks? – Pacerier Mar 9 '18 at 23:10
  • 1
    @Pacerier, Re "Surely there got to be some": not at present, which is why the world needs open firmware and hardware for disks. Benevolent sounding advertising to the contrary, "SMART" is the NSA's playground. – agc Mar 10 '18 at 6:15
1

Sparklines may help... first get the spark util and install the most text viewer.

  1. This hack displays an abstract view of the first 61 sectors of /dev/sda:

    for f in `seq 0 1 60` ; do :
        printf "%.03i %s\n" $f \
            $({   printf '4294967295\n'
                  sudo dd if=/dev/sda skip=$f bs=512 count=1 2> /dev/null |
                  od -v -A n -t uI ; } | 
               spark | sed 's/^.//' )
    done | most
    

    Output on my system (abbreviated first two lines):

    000 ▁▁▁▃▆▆▁▁▄▇▂▆▁▁▆▁▁▁▄▄▄▁ ...
    001 ▁▆▃▇▂▁▁▁▄▃▆▂▄▂▁▁▁▁▁▁▁▂ ...
    

    Each row represents a disk sector. Short lines are small numbers, tall lines are large numbers. Each sparkline char here represents an unsigned integer, (4 bytes), which is relatively compact. If it doesn't all fit on the screen in most, use the right and left arrows to scroll sideways.

    To see what empty data looks like replace the /dev/sda with /dev/zero.

    The above code can be used to skim a disk, just by changing the seq values. Use sudo hdparm -i /dev/sda | grep CurCHS to get the cylinder (tracks), heads, and sector numbers, which on my system returns:

    CurCHS=16383/16/63, CurSects=16514064, LBA=yes, LBAsects=312581808
    

    This means every 16383rd sector is the first sector of a (virtual) track. For a mostly blank disk skimming these would show where the data runs out.

    Changing the seq above to seq 0 16383 $((16383*60)) would show the first sectors of the first 61 tracks on the disk.

    To make the sparkline show just 1 byte per char, change 4294967295 to 255, and uI to uC.


    Notes on how the code works:

    dd sends one 512 byte sector of data to od which outputs unsigned integers, which is the largest number that spark understands.

    spark uses relative heights calculated from the input, (try spark <<< '1 2 3 4' to see how it works), which means unless there's a leading maximum number, the different lines could be to different scales. To avoid misleading different scales, printf is used to prepend the correct maximum which makes spark use one consistent scale, but leaves behind an unwanted first tall char, which is later removed with sed.


  2. A similar technique can show a fuzzy overview of the data density of a disk or partition. Instead of abstracting a few bytes into a spark character, abstract a sampling of blocks, or rather the relative compression ratios of a sampling of blocks.

    Two more utils are required: pigz, (for mostly headerless zlib compression), and pv. pv is really optional, but a progress bar makes the wait less tedious.

    The code:

    d=sda b=512 c=1 m=$(( (512*$(</sys/block/${d/s???*/${d%%[0-9]*}/$d}/size))/b )) \
    s=200 i=$((m/s)) ; \
    for ((f=0;f<m;f+=i)) ; do 
       sudo dd if=/dev/${d} skip=$f bs=$b count=$c 2> /dev/null | 
       pigz -9z | 
       wc -c
    done | pv -l -s $s -i "0.1" \
              -F 'Read %b of '"$((s+1)) $((b*c))-byte"' blocks %p%e' | 
    spark
    

    Output (on my HD):

    Read  201  of 201 512-byte blocks [=====================================>] 100%
    ▇███▁█▂██▃█▆▂▂██▃▆███▆█▅▁█████▇▂▇▂▁▇▂▂▂▆▄█▃▅█▁▄█▃▅▅▅▃▃▂▅
    ▃▂█▆▅▇▁▅▃▆▄▁█▃▇▁▁▂▂▅█████▇█████████▆█████████▅█▁████████
    ███████████████████████▆███████▆██▃███████▁████████▂███▂
    ▁▁▁▁▂▂▂▂██████████▁█▁▁▅▃▁▃█▂▅▄▅▁▁
    

    The hard drive $d is divided into 200 ($s) parts, and the first block of each part is compressed with pigz, piped to wc for a byte count, and the resulting list of numbers is fed to spark.

    The 200 spark characters represent the data densities of 200 512 byte sample blocks. The filled-in chars are blocks with dense data which won't compress, the shorter chars are sparse data which is quite compressible.

    Set s=2000 for a more meaningful view. The variables are all at the beginning, and can be tweaked as desired. Increase $b (but of the form 512*2^n only) or $c to read larger samples. It's interesting that increasing the sample sizes (reading an 8K block instead of 512 byte one) generally draws a very similar sparkline, which implies a 512 byte block is good enough...

  • Pending: a bash array would be a better way of associating magic numbers and strings like 255 and uC... – agc Mar 5 '18 at 9:09
  • This is a nice visualization idea. Where'd you get the idea from? – Pacerier Mar 9 '18 at 16:25
  • "Pending" meaning? – Pacerier Mar 9 '18 at 16:37
  • @Pacerier, Own idea SFAIK, thanks. Pending means I'll revise the answer to add array code with sensible values later. – agc Mar 9 '18 at 20:32
  • Regarding method #2: the current code is bad for large values of $s -- spark is dog-slow if an input list has more than a few thousand numbers. – agc Mar 10 '18 at 3:36
1

dd if=/dev/sda | hd | less

That will output the entire contents of the disk in hexadecimal, one screen full at a time. Long runs of the same byte sequence will be noted and skipped.

  • 1
    hd /dev/sda | less – user1133275 Mar 5 '18 at 1:53
  • @user1133275, oh yea... hd does take a file argument.. I'm just so used to using it as a filter... probably because normally I specify count and seek arguments to dd to only dump specific parts of the drive instead of the whole thing. – psusi Mar 5 '18 at 2:19
  • @psusi, Will this crash if the input is a 4TB disk? Also does it work properly for both ssd and harddisk? – Pacerier Mar 5 '18 at 2:47
  • @Pacerier It will not crash on lage disks, it doesn't matter if it's a HDD or SDD. – Kusalananda Mar 5 '18 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.