3

I have the following bash/awk script, which does what I want, except that it adds a control symbol that I do not know what it means.

#!/usr/bin/env bash

# Merge two cross section files from Resummino

file1=filein1
file2=filein2
fileout=fileout

awk '{
        xs_nlo[$1," ",$2] += $4
        xs_lo[$1," ",$2] += $3
    }
    END {
        for (xs in xs_nlo){
            print(xs, xs_lo[xs], xs_nlo[xs])
        }
    }
' "${file1}" "${file2}"

Possible content of filein1:

100.000 500.000 7.878892e+00 1.027803e+01
100.000 1000.000 9.667085e+00 1.274467e+01
100.000 2000.000 1.029358e+01 1.361803e+01
100.000 5000.000 1.049836e+01 1.390297e+01
100.000 10000.000 1.052944e+01 1.394593e+01

and similar for filein2, except that the last two values in each line differ. The awk script is supposed to add up the third values in each line, assuming the first and the second values are the same. Same for the fourth value.

The output file is supposed to have the same structure as the input files. In the console, the output file looks the same, but opening it in vim, I see a control character that is not easy to search for:

100.000^\ ^\300.000 3.42 4.57283

What does this ^\ ^\ mean and how can I get rid of it?

  • 1
    the source of the problem is your separator $1," ",$2 – RomanPerekhrest Jun 28 '17 at 15:05
  • @RomanPerekhrest Yeah, well, I thought so. So I replaced the " " with "x" and then replaced x with " " in the end again, but I get the same output. How do I fix it? – pfnuesel Jun 28 '17 at 15:19
  • @RomanPerekhrest Okay, I understand now. I was focusing on " ", thinking that the whitespace is the problem, but the only problem was the ,. Thanks for your help! – pfnuesel Jun 28 '17 at 15:24
  • @RomanPerekhrest If you write an answer, no matter how short, I will mark it as solution. – pfnuesel Jun 28 '17 at 15:25
  • ok, added the answer – RomanPerekhrest Jun 28 '17 at 15:37
4

The source of the problem is in separator $1," ",$2.

From documentation:

Multidimensional arrays are supported in awk through concatenation of indices into one string. awk converts the indices into strings (see Conversion) and concatenates them together, with a separator between them. This creates a single string that describes the values of the separate indices. The combined string is used as a single index into an ordinary, one-dimensional array. The separator used is the value of the built-in variable SUBSEP


SUBSEP
The subscript separator. It has the default value of "\034" and is used to separate the parts of the indices of a multidimensional array. Thus, the expression ‘foo["A", "B"]’ really accesses foo["A\034B"]


Standard awk simulates multidimensional arrays by separating subscript values with commas. The values are concatenated into a single string, separated by the value of SUBSEP.
The fact that such a subscript was created in this way is not retained; thus, changing SUBSEP may have unexpected consequences.

1

For the second part of:

What does this ^\ ^\ mean and how can I get rid of it?

Some other answer has already explained that the comma (,) in [$1," ",$2] becomes the SUBSEP. Thta is the character \034 (in octal) or 0x1C (in hex).

That character could also be written in control encoding as ^\.
That is the reason to have the two ^\ in the output.

The solution is to take back (from the awk program) the control of how the indices of the array are created. The most simple solution is to concatenate the two fields ($1 and $2) together, however, that will make equal this two array keys (in no particular language):

$1=abc  $2=def     $1$2 is abcdef
$1=a    $2=bcdef   $1$2 is abcdef

The solution is to use a character that does not appear in the values of $1 or $2. Space may seem like a candidate, but the FS (field separator) is a more general solution which also could separate the fields when printed. Thus, this script works for any FS:

#!/bin/bash

# Merge two cross section files from Resummino

file1=filein1
file2=filein2
fileout=fileout


awk 'BEGIN{OFS=FS} {
          point = $1 FS $2
          xs_nlo[ point ] += $4
          xs_lo [ point ] += $3
     }
     END {
         for (xs in xs_nlo){
             print(xs, xs_lo[xs], xs_nlo[xs])
         }
     }
    ' "${file1}" "${file2}"
  • Thanks, very informative. Too late to mark it as accepted answer. ;-) – pfnuesel Jun 29 '17 at 20:15
0

Replacing xs_nlo[$1," ",$2] with xs_nlo[$1" "$2] fixed it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.