0

I have a problem, I have a file test.txt that has a content like this:

dn: serv=CSPS,mscId=167e48dc2b7a42d4acce611c8b477262,ou=multiSCs,dc=three
structuralObjectClass: CP1
objectClass: CP1
objectClass: CUDBServiceAuxiliary
objectClass: CP2
objectClass: CP3
objectClass: CP4
objectClass: CP5
objectClass: CP6
UNKNLOCDATECS:: FQsJ
UNKNLOCDATEPS:: FgMe
ISTTIMESTAMP:: FgMIDyI7
CSULTIME:: HgMWCzYo
CSLOCTIME:: AQQWBA0R
PSULTIME:: HgMWDBco
PSLOCTIME:: HgMWDBco
SCHAR:: AgA=
ICS: 1
CAT: 10
DBSG: 1
OFA: 1
SOCB: 1
PWD: 0000
PWDC: 0
SOCFB: 0

Every time the text CSULTIME:: and CSLOCTIME:: are found I want to replace the value after those literals with the following funtion to decode that timestamp to a recognize format (if I can replace both values in a single scan of the file better as we are talking about 8 GB file and the function is the same in both cases):

base64 -d | hexdump -v -e '1/1 "%02d" ' | awk 'BEGIN {FS = ""} {print "20" $5 $6 "-" $3 $4 "-" $1 $2 " " $7 $8 ":" $9 $10 ":" $11 $12}'

If I do in unix an echo for those two values:

For CSULTIME the result would be 2022-03-30 11:54:40: echo -n "HgMWCzYo" | base64 -d | hexdump -v -e '1/1 "%02d" ' | awk 'BEGIN {FS = ""} {print "20" $5 $6 "-" $3 $4 "-" $1 $2 " " $7 $8 ":" $9 $10 ":" $11 $12}'
For CSLOCTIME the result would be 2022-04-01 04:13:17: echo -n "AQQWBA0R" | base64 -d | hexdump -v -e '1/1 "%02d" ' | awk 'BEGIN {FS = ""} {print "20" $5 $6 "-" $3 $4 "-" $1 $2 " " $7 $8 ":" $9 $10 ":" $11 $12}'

So at the end the file would have these values for CSULTIME and CSLOCTIME:

dn: serv=CSPS,mscId=167e48dc2b7a42d4acce611c8b477262,ou=multiSCs,dc=three
    structuralObjectClass: CP1
    objectClass: CP1
    objectClass: CUDBServiceAuxiliary
    objectClass: CP2
    objectClass: CP3
    objectClass: CP4
    objectClass: CP5
    objectClass: CP6
    UNKNLOCDATECS:: FQsJ
    UNKNLOCDATEPS:: FgMe
    ISTTIMESTAMP:: FgMIDyI7
    CSULTIME:: 2022-03-30 11:54:40
    CSLOCTIME:: 2022-04-01 04:13:17
    PSULTIME:: HgMWDBco
    PSLOCTIME:: HgMWDBco
    SCHAR:: AgA=
    ICS: 1
    CAT: 10
    DBSG: 1
    OFA: 1
    SOCB: 1
    PWD: 0000
    PWDC: 0
    SOCFB: 0

I'm completely lost as all combinations I'm trying of sed I can't make them work.

Thanks in advance!!!!

5
  • Just a thought: Once the command line to sed become more than I can read in one go, or I need two sed commands - I switch to perl.
    – White Owl
    Apr 28 at 14:20
  • I have problems with sed, imaging with perl :) But of course, if I can do the same I don't mind if is with perl Apr 28 at 14:22
  • 1
    Didn't you ask essentially the same thing before? Replace a base64 value in a file in unix Apr 28 at 23:11
  • For anything even moderately complex, Perl is actually a lot easier to work with than sed. And certainly easier than a pipeline of multiple programs run from shell.
    – cas
    Apr 29 at 1:00
  • steeldriver7, I tried that solution, the problem is that these values are encoded Base64 to Hex to Decimal and that solution doesn't return a timestamp, just some weird characters Apr 29 at 7:08

3 Answers 3

1

I'd do something like:

perl -MMIME::Base64 -pe 's{^[^:]*TIME:\K: (\S+)}{
  my ($d, $m, $y, @t) = unpack "C*", decode_base64 $1;
  sprintf "20%02d-%02d-%02d %02d:%02d:%02d", $y, $m, $d, @t}e'
2
  • Thanks a lot, I will try this on Tuesday (I already logged out of work and Monday is bank holiday in Ireland). I'm sure this will be a lot faster than two seds Apr 29 at 14:54
  • Definitely a better option than mine with the sed, as I can do all the changes in a single scan of the file and a lot faster. Thanks Stephane!!!!! May 4 at 16:04
0

You can use the following method where we first setup the paraphernalia for base64 decoding and then using GNU sed to process the encoded data.

#------------------
#  base64 in sed
#------------------
set -u

#> present century
century=$(date '+%C')

#> format string printf
fmt='%s%s\n'

#> bit, sextet, & octet regex
bit='[01]'
sextet="${bit}{6}"
 octet="${bit}{8}"; byte=$octet

#> base64 charset
b64='/[:alnum:]+'
declare -a b64_chars=({A..Z} {a..z} {0..9} + /)

#> user-defined helper function(s)
oneLine() {
  # collapse stdin into one line
  paste -sd'\0' -
}

esc_rhs() {
  # make stdin pkuggable
  # on the rhs of a s///
  sed -e '
    s:[\/&]:\\&:g
    $!s:$:\\:
  ' -
}

dec2bin() {
  # create an array of size 2^$1
  # Usage: dec2bin 6
  # creates array d2b as:
  # underscore for clarity only
  # $d2b[0] => "000_000"
  # $d2b[1] => "000_001"
  #        ...
  # $d2b[63] => "111_111"

  eval "d2b=($(yes '{0,1}' | sed "$1q" | oneLine))"
}

#> build the encoding lookup table
encode_tbl=$(printf '%s\n' "$(
dec2bin 6
i=0
for c in "${b64_chars[@]}"
do
  printf "$fmt" "$c" "${d2b[$i]}"
  (( i++ ))
done
)" | esc_rhs)

#> build the decoding lookup table
decode_tbl=$(printf '%s\n' "$(
dec2bin 8
for dec in {0..255}
do
  hex=$(printf '%02x' "$dec")
  bin=${d2b[$dec]}
  printf "$fmt" "$bin" "$hex"
done
)" | esc_rhs)

#> hex to bin
h2d=$(for i in {0..255};do
  printf 'x%02x:%02d\n' "$i" "$i"
done | oneLine | esc_rhs)

#######> main()
sed -E "
  s/^\s*(CSULTIME|CSLOCTIME)::\s*/&\n/;T
  h;s/\n.*//
  x;s/.*\n//
  s/[^${b64}]//g
  /^\$/d;s/^|\$/\n/g

  # unencode
  s/\$/$encode_tbl/
  :encode
  /\n\n/!{
    s/((\n)[${b64}])(.*\1(${sextet}))/\4\2\3/
    b encode
  }

  s/\n.*//;:pad
  /^(${octet})+\$/!{
    s/\$/0/;b pad
  }
  # chunk it in 8-bit portions
  s/(${octet})/& /g
  G;s/\$/${decode_tbl}/;t decode
  :decode
    s/(${octet}) (.*\n\1([[:xdigit:]]{2}))/\3\2/
  t decode
  s/\n.*//

  #> hex to decimal
  s/^|\$/\n/g
  s/\$/$h2d/;t hex2int
  :hex2int
  s/\n([[:xdigit:]]{2})(.*\n.*x\1:([[:digit:]]+))/\3\n\2/
  t hex2int
  s/\n.*//

  #> rearrange decoded value into
  #> yyy-mm-dd hh:mm:ss format
  s/(..)(..)(..)/\3\2\1/
  s/../&-/g
  s/(.*)./$century\1/
  s/-/:/3g
  s/:/ /

  #> stitch back the prefixes
  #> CSULTIME, CSLOCTIME
  H;z;x;s/\n//
" file

Output:-

-1e-0===
dn: serv=CSPS,mscId=167e48dc2b7a42d4acce611c8b477262,ou=multiSCs,dc=three
structuralObjectClass: CP1
objectClass: CP1
objectClass: CUDBServiceAuxiliary
objectClass: CP2
objectClass: CP3
objectClass: CP4
objectClass: CP5
objectClass: CP6
UNKNLOCDATECS:: FQsJ
UNKNLOCDATEPS:: FgMe
ISTTIMESTAMP:: FgMIDyI7
CSULTIME:: 2022-03-30 11:54:40
CSLOCTIME:: 2022-04-01 04:13:17
PSULTIME:: HgMWDBco
PSLOCTIME:: HgMWDBco
SCHAR:: AgA=
ICS: 1
CAT: 10
DBSG: 1
OFA: 1
SOCB: 1
PWD: 0000
PWDC: 0
SOCFB: 0

0

I got it, I was making it right (with sed, don't know how to use hexdump in perl) the problem were the characters I needed to escape inside the sed.

Now it works perfectly with this, although I will have to do two seds:

sed  's/\(CSULTIME::\)\(.*\)/echo -n \1" ";echo \2\| base64 -d \| hexdump -v -e '\''1\/1 "%02d" '\'' \| awk '\''BEGIN {FS = ""} {print "20" $5 $6 "-" $3 $4 "-" $1 $2 " " $7 $8 ":" $9 $10 ":" $11 $12}'\'' /ge ' 
sed  's/\(CSLOCTIME::\)\(.*\)/echo -n \1" ";echo \2\| base64 -d \| hexdump -v -e '\''1\/1 "%02d" '\'' \| awk '\''BEGIN {FS = ""} {print "20" $5 $6 "-" $3 $4 "-" $1 $2 " " $7 $8 ":" $9 $10 ":" $11 $12}'\'' /ge '

#------------

You can factor away your code in a bash function, export it, then have it called within the GNU sed 's s///e

fx() {
  printf '%s' "$2" | base64 -d |
  hexdump -v -e '1/1 "%02d" ' |
  awk -v temp="$1" 'BEGIN {FS = ""} {print temp "20" $5 $6 "-" $3 $4 "-" $1 $2 " " $7 $8 ":" $9 $10 ":" $11 $12}'
}

export -f fx

sed -E "s/((CSULTIME|CSLOCTIME)::\s*)(\S.*)/fx '\1' '\3'/e" file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.