3

I have a text file with lines (each of 631 characters) consisting of 16 hexadecimal numbers 0,1,2,...,9,A,B,...,F. I would like to replace each number with the corresponding 4 bits of 0 and 1, i.e. 0 with 0000, 1 with 0001, ..., F with 1111 and then save the file as .bin. How can I do so in Linux terminal?

Some lines of my text file are:

85868D6B0FD5F3186DE92A7FC023CC848110FE61E3045B24BD905C8FDF34698794718CB2BAFF2AFC488C07B01510646141290A8FE55BF41F9D5033B13B31473FB4F7E199624370DC9C7587E5FB84B209B104B2BC194243EFC77C479BEA6778C73FB55939CB8654B1EB0B597CEA9507FE5A9636229BE0E7ECA27844D3A4AE16096D9DDC125728A1BF0C3623518311CF7CE629C15EA767E1F6C992106E86D962D47E307AE9F71FE5C6B664820E3D7521D0F783913E8A116BFD42C65FD2E28C9114CF4BFCB8A2AFD9979E7B5EBC7B59C7A31A7C12931D2C794EF5AED991B32560BBEFCC82D3A6AC6D371C7AC6C06DED6BCEFFC3D9E11A17B4738EA2ECF8CDD14805E57C50F973C6832DB94517A71C06E18329694B8CB7F7028730E54F74E27EF35B3D776A7E46A3F187691D460CBFBCD14BAC4AED32E5592AFDAEC5206
6321EB4F15D5F01D2A5F7474AC6FFCD1A8B52E680AC52CA9B500C956FD269DB8107EFCF6EC0B23B2C54B3A727CC53F6793019BF0D95A037485413576E4A0172A597027ED71D6E785430FA4748E164E5290214B01740736C0EC79D286692170B6EFBA21592EC485350F0DC6C635CD3FA7DAED258405C4247BFA68613B5F7CBEC903E16EA2E780D0934F0A2918F837CF0115B3D76AAFD3F37D905F9572D334D959DDD22C37EBA7707A76333A491F79B9C53A7912AEA38ED400949EB9AD9FCEE71765F4F4A2E21C481A509AB6807DDF5374D9A1A4D0C589E628CAA3C6DC9E4A5C1A7EF3C36A4D2231E2C1E856A19B6F7F4DE713D5BE8A4EFA750FB29A34F00754D875C22C2FC981FAECCC7113D1B0D6FF3BCC747852789A662D104DF1A8C90EB4DDFA48502A3BA9AAE70E7229E8A3D6FBBD00C19D99765BAFC9B09D547
2F75AC10A9E20C07DD4686C5706F66EB1AB33ED9DEDAD69079B6EC0C24C8DF24CB1855825D80CD484E5623BB246041D29240720500EA87586CC4CE5A00990EF8B5B7E2A3CF3E997323870499451E7BFA7F23A2208523B84A98DDC0763701C1AB292D854ADA8FB2E060088D7B3436571745DF25182D7F5A646A6F7DD830B9CCC12C37D416882DD7AAA0E8CB2CE50CE33AD1674C0E78E0F171DEC2D087C6CF2E711FAC2548CCB8C0312D59055A0A3A145B402A749F768F637549E0749215B5542BBC9D07EF9416C1BAB1FBEAC4709C458E7C266342908421BC8F67B7E21ED228D13A0F2750C290EFF8CAF7D2AD6D6C2CB12B81570190E1BC555E0CFBA4951BEAD3740D3BEBF2C7BF2AB64C7F046E5B865D85A1411F854906C05785A0A3684EC0F69D9BAB047A9EC17F973E78A92838C73270F97E54CEF4FABAA82876F
  • 1
    This question has an answer HERE – MelBurslan Apr 27 '16 at 14:24
  • 1
    for n in $(cat myfile | tr 'a-f' 'A-F');do echo "obase=2; ibase=16; $n" | bc;done then you may need to do some padding for binary numbers shorter than 4 digits – MelBurslan Apr 27 '16 at 14:33
  • @MelBurslan +1 and you should write that as an answer. – dr01 Apr 27 '16 at 14:34
1

Posting my comment as an answer to be marked as "solved"

for n in $(cat myfile | tr 'a-f' 'A-F');do echo "obase=2; ibase=16; $n" | bc ;done

Okay, in the light of what you have said and what I understood, I have a couple of suggestions to you. The first one, converts each hex line into a new binary line and puts a line break at the end of each line. here is that solution:

cat myfile | while read line
do
  len=${#line}
  binline=""
  index=1
  while [ ${index} -le ${len} ]
  do
    hex=$(echo ${line} | cut -c ${index} | tr 'a-f' 'A-F')
    bin=$(echo "obase=2; ibase=16; ${hex}"|bc)
    padbin=$(printf "%04d" ${bin})
    binline="${binline}${padbin}"
    (( index++ ))
  done
  echo ${binline} >> program.bin
done

And, if I am not mistaken, you are trying to use this binary file as a program to an embedded system or something similar. If that is the case the line breaks may present a problem. If this is the case, i.e., you want the whole file as one single humongous line, you can use the following snippet:

cat myfile | while read line
do
  len=${#line}
  index=1
  while [ ${index} -le ${len} ]
  do
     hex=$(echo ${line} | cut -c ${index} | tr 'a-f' 'A-F')
     bin=$(echo "obase=2; ibase=16; ${hex}"|bc)
     printf "%04d" ${bin} >> p2.bin
     (( index++ ))
  done
done

Having said all of this, if your hexadecimal values file is in gigabytes size or even few hundred megabytes size, bash will complain and may throw a cryptic error. So, it is in your best interest to chop this input file to more manageable chunks and process them piece-by-piece.

Please note that, I wrote these scripts in the simplest for to understand. Hence it will have some performance penalties when you run them. 3 lines takes about 6 seconds of processing time. It is up to you to optimize it, if you find the wait intolerable.

  • 1
    Thank you ! And how can I save my file in .bin format? – user166316 Apr 27 '16 at 14:40
  • I am not sure what you mean by bin format but you can output these numbers to a files named as program.bin for instance using redirection at the end of the command, i.e., for n in $(cat myfile | tr 'a-f' 'A-F');do echo "obase=2; ibase=16; $n" | bc ;done >program.bin, but since you mentioned the word program, I am assuming there has to be some other heading or trailing blocks for your machine to understand, this is a program, not a random string of 1s and 0s. At this point you are on your own. – MelBurslan Apr 27 '16 at 14:46
  • OK, I got it ! Thanks ! But the terminal closes for my large file of 58 GB when I run the for loop. What can I do? – user166316 Apr 27 '16 at 15:32
  • I tried for a small file, what I see in the output file is: 10000101100001101000110101101011000011111101010111110011000110000110\ 11011110100100101010011111111100000000100011110011001000010010000001\ ... After every 68 zeros and ones there is a backslash, how can I remove it and have all 0s and 1s one after the other? – user166316 Apr 27 '16 at 15:49
  • could you please post a few lines from your original file, so that I can see and test my code snippet against it ? Please update your original post and put the lines there, not here as a comment. – MelBurslan Apr 27 '16 at 15:59
2

xxd -r -p

Example:

sudo apt install xxd
printf 12345678 | xxd -r -p | od -A n -t x1 -v

gives:

 12 34 56 78

od dumped the bytes, so xxd -r -p did what we wanted.

Tested on Ubuntu 19.04, xxd V1.10.

0

You can use printf:

$ printf '%04d\n' 0
0000

$ printf '%04d\n' 1
0001

Or the POSIX Awk stdlib library:

$ awklib 'BEGIN {print base_conv("F", 16, 2)}'
1111

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.