1

I have 487 bytes fixed length file and each record position 407-415 (9 bytes) is a numeric field. Sometimes we are receiving combination alpha and numeric characters in that filed (123ABC123,ABC000123....). In case of record contains like this, we want to replace with 00000000 in those bytes.

2
  • 1
    Is each file a single record? Is it a text or binary file? The usual tool for doing surgery on this kind of fixed record file is dd but if it is text then sed would be a fine tool to use?
    – icarus
    Jun 11, 2019 at 4:18
  • 1
    Could you please provide a minimal example of an input file, and the expected output file? Could you please include examples of what should be changed and what should not?
    – Sparhawk
    Jun 11, 2019 at 4:20

3 Answers 3

0

And an awk solution

awk 'BEGIN{FS=OFS=""}{f=0;for(i=407;i<=415;i++){if($i!~/0-9/){f=1}};if(f){for(i=407;i<=415;i++){$i=0}};print }' fixed

We define the field separator to be empty, every character is a field. Then we set a flag f for each record to be 0 (assume numeric). We compare characters 407-415 against against /0-9/. If any don't match we set f to 1. Then, if f=1, we replace all 9 characters with 0s and print the line.

0

As those other answers look complicated to me, here is one simple sed one-liner:

sed -E '/^.{406}[0-9]{9}/!s/(.{406}).{9}/\1000000000/'

The extended (option -E) regular expression ^.{406}[0-9]{9} matches lines (I assume one record is one line, otherwise add option -z to your GNU sed), that start (^) with 406 random characters (.{406}) followed by nine numerical characters, thus 407 to 415 are all digits, so for the invers match (!) the first 406 chars remain (\1 in the replacement stands for the first () part) and the following nine chars get replaced by zeroes.

Please note, that as you ask for text processing, this is text processing and not byte processing. Multi-Byte characters are counted as one char, not two bytes

3
  • One record being one line is a big assumption.
    – icarus
    Jun 11, 2019 at 5:45
  • @icarus The question is tagged /linux, so it's GNU sed and you can simply add option -z to treat the whole file instead of line-by-line. For non-GNU sed, start the script with the usual H;1h;$!d;x
    – Philippos
    Jun 11, 2019 at 5:55
  • -z works as long as there are no NUL characters in the file. In my reading of the question the OP didn't explicitly ask for text processing, although you can argue that by giving sed and awk tags this is implied. Your answer is pretty much exactly what I would suggest if the file is known to be text. My experience with fixed record formats tells me these are often binary.
    – icarus
    Jun 11, 2019 at 6:09
0

this should work, no matter if your input file is text or binary:

printf %s "$(head -c 406 test_file | xxd -p)" "000000000000000000" "$(tail -c $((487-415)) test_file | xxd -p)" | tr -d '\n' | fold -32 | xxd -p -r -c 16 > test_file_final

or this one:

printf %s "$(head -c 406 test_file | xxd -p)" "$(head -c 9 /dev/zero | xxd -p)" "$(tail -c $((487-415)) test_file | xxd -p)" | tr -d '\n' | fold -32 | xxd -p -r -c 16 > test_file_final

You don't need to use: $((487-415))... you can simply use 72 instead... I made it that way, to make more obvious were the numbers came from.

What you are doing with this, is taking the first 406 bytes of the file, then 9 null bytes ( this 9 null bytes, could be generated, for example, from a plain string with 18 zeros on it [first option] or with 9 raw bytes from /dev/zero [second option] ) then from byte 416 of the file, till the end, and then you concatenate this tree chunks together, building your desired output.


Or, if you want to replace with the zero char (not the null byte), use this one:

printf %s "$(head -c 406 test_file | xxd -p)" "303030303030303030" "$(tail -c $((487-415)) test_file | xxd -p)" | tr -d '\n' | fold -32 | xxd -p -r -c 16 > test_file_final

Commands above, will replace, no matter what: no condition is tested. If you want to test for HEX chars in those bytes, use this to replace with null bytes:

One-liner version:

if ! [[ "$(tail -c $((487-406)) test_file | head -c 9)" =~ [[:xdigit:]]{9} ]]; then printf %s "$(head -c 406 test_file | xxd -p)" "000000000000000000" "$(tail -c $((487-415)) test_file | xxd -p)" | tr -d '\n' | fold -32 | xxd -p -r -c 16 > test_file_final; fi

Script version:

if ! [[ "$(tail -c $((487-406)) "$1" | head -c 9)" =~ [[:xdigit:]]{9} ]]
then
    printf %s "$(head -c 406 "$1" | xxd -p)" \
              "000000000000000000" \
              "$(tail -c $((487-415)) test_file | xxd -p)" \
              | tr -d '\n' \
              | fold -32 \
              | xxd -p -r -c 16 \
              > "${1}_with_replacement_done"
fi

or this, to replace with zero chars:

One-liner version:

if ! [[ "$(tail -c $((487-406)) test_file | head -c 9)" =~ [[:xdigit:]]{9} ]]; then printf %s "$(head -c 406 test_file | xxd -p)" "303030303030303030" "$(tail -c $((487-415)) test_file | xxd -p)" | tr -d '\n' | fold -32 | xxd -p -r -c 16 > test_file_final; fi

Script version:

if ! [[ "$(tail -c $((487-406)) "$1" | head -c 9)" =~ [[:xdigit:]]{9} ]]
then
    printf %s "$(head -c 406 "$1" | xxd -p)" \
              "303030303030303030" \
              "$(tail -c $((487-415)) test_file | xxd -p)" \
              | tr -d '\n' \
              | fold -32 \
              | xxd -p -r -c 16 \
              > "${1}_with_replacement_done"
fi
3
  • 2
    These unconditionally change the field to zero, whilst the OP asked for it to be changed to zeros if it was an alpha-numeric field. The implication is that if it was numeric then the field should be left unchanged.
    – icarus
    Jun 11, 2019 at 4:47
  • that is true!; I didn't notice that the OP asked for a conditional replacement. This will replace no matter what. Jun 11, 2019 at 4:50
  • fixed!, now, it'll do a conditional replacement. Jun 11, 2019 at 11:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .