5

After parsing the input line, awk provides access to the original line ($0) as well as to each individual column ($1, $2, ...). While performing this process (lazily, on demand) - it knows exactly the position of the character where the 2nd column starts.

  1. Does it provide access to this info (i.e., at what position in the original line $0 does the 2nd column start)?
  2. If not - is there any sane/elegant way of finding it out properly? (I'm about to start coding an ugly and inefficient way of mimicking awk's internal behavior by using dynamic-regexps based on FS, handling special FS==" " case, using capturing groups, etc. But wanted your advice before I dive deep into it.)

Example 1 (default FS):

$ echo -n -e " \tFirst \t\t  Second \t Third  \t"\
|awk -F" " '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'\
|sed 's/\t/\\t/g'

FS:[ ]
$0:[ \tFirst \t\t  Second \t Third  \t]
$1:[First]
$2:[Second]
$3:[Third]

in here - I need to know that the 2nd column (Second) starts with the letter S and this is the 13th character in the input line (so I would be able to store First as the key, and preserve/store the Second \t Third \t intact as the value for the further use)


Example 2 (TAB as FS):

$ echo -n -e " \tFirst \t\t  Second \t Third  \t"\
|awk -F"\t" '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'\
|sed 's/\t/\\t/g'

FS:[\t]
$0:[ \tFirst \t\t  Second \t Third  \t]
$1:[ ]
$2:[First ]
$4:[  Second ]
$5:[ Third  ]

in here - I need to know that the 2nd column (First) starts with the letter F and this is the 3rd character in the input line - so I would be able to store (space) as the key, and preserve/store First \t\t Second \t Third \t intact as the value for the further use


Example 3 (custom FS):

$ echo -n -e " \tFirst \t\t  Second \t Third  \t"\
|awk -F"[ \t]+" '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'\
|sed 's/\t/\\t/g'

FS:[[ \t]+]
$0:[ \tFirst \t\t  Second \t Third  \t]
$2:[First]
$3:[Second]
$4:[Third]

in here - I need to know that the 2nd column (First) starts with the letter F and this is the 3rd character in the input line - so I would know the 1st column is an empty string, and store the First \t\t Second \t Third \t as the value for the further use


Example 4 (complex FS):

$ echo "-11...22;,;..;33-44...;"\
|awk -F"[^0-9-]+" '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'

FS:[[^0-9-]+]
$0:[-11...22;,;..;33-44...;]
$1:[-11]
$2:[22]
$3:[33-44]

in here - I need to know that the 2nd column (22) starts with the character 2 and this is the 7th character in the input line - so I would be able to store -11 as the key, and 22;,;..;33-44...; as the value for the further use


Basically the idea is to grab some (1st) columns for a custom use and to preserve (store into a variable) the remainder of the line (from 2nd column till end of line) intact.

3 Answers 3

2

With GNU awk for the 4th arg to split():

$ cat tst.awk
{
    split($0,flds,FS,seps)
    key = flds[1]
    pos = length(seps[0] flds[1] seps[1]) + 1
    val = substr($0,pos)
    printf "key=<%s>\npos=<%s>\nval=<%s>\n\n", key, pos, val
}

$ printf -- ' \tFirst \t\t  Second \t Third  \t\n' | awk -f tst.awk
key=<First>
pos=<13>
val=<Second      Third          >

$ printf -- '-11...22;,;..;33-44...;\n' | awk -F'[^0-9-]+' -f tst.awk
key=<-11>
pos=<7>
val=<22;,;..;33-44...;>
2
  • perfectly works, upvoted, but accepted another solution because of performance - see my separate answer, thanks
    – Vlad
    May 13, 2021 at 5:04
  • You specifically asked for a sane/elegant way, not the fastest way possible and it's OK if it only works in specific cases. @αғsнιη had provided the fastest solution but deleted it when I pointed out that it only works in specific cases but they should probably undelete it as that may be good enough for you. I see you had posted an answer that's since been deleted so I can't comment on it directly but I will say that those results you posted may have been affected by cache-ing and will be affected by the awk version you're running (not all awks only split up to the first referenced field).
    – Ed Morton
    May 13, 2021 at 13:14
2

In GNU/awk, you can use split() with the optional seps argument, then iterate over array and seps to generate start and size arrays for each field, accumulating the length of each field and separator.

This code is for GNU/awk. The function Offsets() takes a text string and a pattern for the field separators, and returns a pair of arrays containing the start column and length of the fields.

$ cat myCols
#! /bin/bash

myCols () {

    local Awk='
BEGIN { cmdDu = "od -A n -t a"; }
#.. Debug the input.
function Debug (tx, Local ) {
    printf ("\nLine %2d: %s\n", NR, tx);
    printf ("%s", tx) | cmdDu; close (cmdDu);
}
#.. Return arrays of column start and length.
function Offsets (col, lth, tx, re, Local, fld, sep, f) {
    delete col; delete lth;
    split (tx, fld, re, sep);
    c = length (sep[0]);
    for (f = 1; f in fld; ++f) {
        col[f] = 1 + c; lth[f] = length (fld[f]);
        c += length (fld[f]) + length (sep[f]);
    }
}
#.. Find fields and show the results.
function Fields (tx, re, Local, col, lth, f) {
    Offsets( col, lth, tx, re);
    for (f = 1; f in col; ++f) {
        printf ("Field %d col %3d lth %3d >%s<\n",
            f, col[f], lth[f], substr (tx, col[f], lth [f]));
    }
}
{ Debug( $0); }
NR == 1 { Fields( $0, ",[[:space:]]*"); }
NR == 2 { Fields( $0, FS); }
'
    awk -f <( printf '%s' "${Awk}" )

}

    {
        echo "Never,   Ever,  Forget,  but maybe,   Forgive"
        echo -n -e " \tFirst \t\t  Second \t Third  \t"
    } | myCols

And test it:

$ ./myCols

Line  1: Never,   Ever,  Forget,  but maybe,   Forgive
   N   e   v   e   r   ,  sp  sp  sp   E   v   e   r   ,  sp  sp
   F   o   r   g   e   t   ,  sp  sp   b   u   t  sp   m   a   y
   b   e   ,  sp  sp  sp   F   o   r   g   i   v   e
Field 1 col   1 lth   5 >Never<
Field 2 col  10 lth   4 >Ever<
Field 3 col  17 lth   6 >Forget<
Field 4 col  26 lth   9 >but maybe<
Field 5 col  39 lth   7 >Forgive<

Line  2:    First         Second     Third      
  sp  ht   F   i   r   s   t  sp  ht  ht  sp  sp   S   e   c   o
   n   d  sp  ht  sp   T   h   i   r   d  sp  sp  ht
Field 1 col   3 lth   5 >First<
Field 2 col  13 lth   6 >Second<
Field 3 col  22 lth   5 >Third<
$ 

If you want the exact original string starting from (say) the third field, use:

if (3 in col) Tail = substr (tx, col[3]);
2
  • 1
    perfectly works, upvoted, but accepted another solution because of performance - see my separate answer, thanks
    – Vlad
    May 13, 2021 at 5:04
  • @Vlad That's OK - I will file mine as a generic code. Maybe useful when reporting data errors, for example if I want to underline the field. May 13, 2021 at 10:10
1

If you're always going to use $1 as the key and the rest of the line (from $2) as the value, then you could use index to find where they are in $0:

  1. Use index to get the location of $1 in $2
  2. Then use the length of $1 to get the substring of $0 where the first position where $2 can be (for cases where $1 can contain a copy of $2)
  3. Then use index again to get the location of $2, so that we can extract the substring starting at $2.

Example:

# foo.awk
function mysplit(array) {
    pos1 = index($0, $1)
    sub1 = substr($0, pos1 + length($1))
    pos2 = index(sub1, $2)
    sub2 = substr(sub1, pos2)
    array[$1] = sub2
}

{mysplit(arr)}

END {
    for (i in arr) {
        printf "[%s]:   |%s|\n", i, arr[i]
    }
}

With your examples:

% echo -n -e " \tFirst \t\t  Second \t Third  \t" | awk -f foo.awk -F " " | sed 's/\t/\\t/g'
[First]:   |Second \t Third  \t|
% echo -n -e " \tFirst \t\t  Second \t Third  \t" | awk -f foo.awk -F "\t" | sed 's/\t/\\t/g'
[ ]:   |First \t\t  Second \t Third  \t|
% echo -n -e " \tFirst \t\t  Second \t Third  \t" | awk -f foo.awk -F "[ \t]+" | sed 's/\t/\\t/g'
[]:   |First \t\t  Second \t Third  \t|
1
  • will not work for some weird/extreme/uncommon field separators - echo AbAAbB | awk -f foo.awk -FAb will output []: |AbAAbB| (instead of []: |AAbB|), but is more-than-good for all my possible use-cases, thanks!
    – Vlad
    May 13, 2021 at 4:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.