AWK: how can I tell where column begins

Question

After parsing the input line, awk provides access to the original line ($0) as well as to each individual column ($1, $2, ...). While performing this process (lazily, on demand) - it knows exactly the position of the character where the 2nd column starts.

1. Does it provide access to this info (i.e., at what position in the original line $0 does the 2nd column start)?
2. If not - is there any sane/elegant way of finding it out properly? (I'm about to start coding an ugly and inefficient way of mimicking awk's internal behavior by using dynamic-regexps based on FS, handling special FS==" " case, using capturing groups, etc. But wanted your advice before I dive deep into it.)

---

Example 1 (default FS):

$ echo -n -e " \tFirst \t\t  Second \t Third  \t"\
|awk -F" " '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'\
|sed 's/\t/\\t/g'

FS:[ ]
$0:[ \tFirst \t\t  Second \t Third  \t]
$1:[First]
$2:[Second]
$3:[Third]

in here - I need to know that the 2nd column (Second) starts with the letter S and this is the 13th character in the input line (so I would be able to store First as the key, and preserve/store the Second \t Third \t intact as the value for the further use)

---

Example 2 (TAB as FS):

$ echo -n -e " \tFirst \t\t  Second \t Third  \t"\
|awk -F"\t" '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'\
|sed 's/\t/\\t/g'

FS:[\t]
$0:[ \tFirst \t\t  Second \t Third  \t]
$1:[ ]
$2:[First ]
$4:[  Second ]
$5:[ Third  ]

in here - I need to know that the 2nd column (First) starts with the letter F and this is the 3rd character in the input line - so I would be able to store (space) as the key, and preserve/store First \t\t Second \t Third \t intact as the value for the further use

---

Example 3 (custom FS):

$ echo -n -e " \tFirst \t\t  Second \t Third  \t"\
|awk -F"[ \t]+" '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'\
|sed 's/\t/\\t/g'

FS:[[ \t]+]
$0:[ \tFirst \t\t  Second \t Third  \t]
$2:[First]
$3:[Second]
$4:[Third]

in here - I need to know that the 2nd column (First) starts with the letter F and this is the 3rd character in the input line - so I would know the 1st column is an empty string, and store the First \t\t Second \t Third \t as the value for the further use

---

Example 4 (complex FS):

$ echo "-11...22;,;..;33-44...;"\
|awk -F"[^0-9-]+" '{print "FS:["FS"]";for(i=0;i<=5;i++)if(""!=$i)print "$"i":["$i"]"}'

FS:[[^0-9-]+]
$0:[-11...22;,;..;33-44...;]
$1:[-11]
$2:[22]
$3:[33-44]

in here - I need to know that the 2nd column (22) starts with the character 2 and this is the 7th character in the input line - so I would be able to store -11 as the key, and 22;,;..;33-44...; as the value for the further use

---

Basically the idea is to grab some (1st) columns for a custom use and to preserve (store into a variable) the remainder of the line (from 2nd column till end of line) intact.

Answer 1

With GNU awk for the 4th arg to split():

$ cat tst.awk
{
    split($0,flds,FS,seps)
    key = flds[1]
    pos = length(seps[0] flds[1] seps[1]) + 1
    val = substr($0,pos)
    printf "key=<%s>\npos=<%s>\nval=<%s>\n\n", key, pos, val
}

$ printf -- ' \tFirst \t\t  Second \t Third  \t\n' | awk -f tst.awk
key=<First>
pos=<13>
val=<Second      Third          >

$ printf -- '-11...22;,;..;33-44...;\n' | awk -F'[^0-9-]+' -f tst.awk
key=<-11>
pos=<7>
val=<22;,;..;33-44...;>

Answer 2

In GNU/awk, you can use split() with the optional seps argument, then iterate over array and seps to generate start and size arrays for each field, accumulating the length of each field and separator.

This code is for GNU/awk. The function Offsets() takes a text string and a pattern for the field separators, and returns a pair of arrays containing the start column and length of the fields.

$ cat myCols
#! /bin/bash

myCols () {

    local Awk='
BEGIN { cmdDu = "od -A n -t a"; }
#.. Debug the input.
function Debug (tx, Local ) {
    printf ("\nLine %2d: %s\n", NR, tx);
    printf ("%s", tx) | cmdDu; close (cmdDu);
}
#.. Return arrays of column start and length.
function Offsets (col, lth, tx, re, Local, fld, sep, f) {
    delete col; delete lth;
    split (tx, fld, re, sep);
    c = length (sep[0]);
    for (f = 1; f in fld; ++f) {
        col[f] = 1 + c; lth[f] = length (fld[f]);
        c += length (fld[f]) + length (sep[f]);
    }
}
#.. Find fields and show the results.
function Fields (tx, re, Local, col, lth, f) {
    Offsets( col, lth, tx, re);
    for (f = 1; f in col; ++f) {
        printf ("Field %d col %3d lth %3d >%s<\n",
            f, col[f], lth[f], substr (tx, col[f], lth [f]));
    }
}
{ Debug( $0); }
NR == 1 { Fields( $0, ",[[:space:]]*"); }
NR == 2 { Fields( $0, FS); }
'
    awk -f <( printf '%s' "${Awk}" )

}

    {
        echo "Never,   Ever,  Forget,  but maybe,   Forgive"
        echo -n -e " \tFirst \t\t  Second \t Third  \t"
    } | myCols

And test it:

$ ./myCols

Line  1: Never,   Ever,  Forget,  but maybe,   Forgive
   N   e   v   e   r   ,  sp  sp  sp   E   v   e   r   ,  sp  sp
   F   o   r   g   e   t   ,  sp  sp   b   u   t  sp   m   a   y
   b   e   ,  sp  sp  sp   F   o   r   g   i   v   e
Field 1 col   1 lth   5 >Never<
Field 2 col  10 lth   4 >Ever<
Field 3 col  17 lth   6 >Forget<
Field 4 col  26 lth   9 >but maybe<
Field 5 col  39 lth   7 >Forgive<

Line  2:    First         Second     Third      
  sp  ht   F   i   r   s   t  sp  ht  ht  sp  sp   S   e   c   o
   n   d  sp  ht  sp   T   h   i   r   d  sp  sp  ht
Field 1 col   3 lth   5 >First<
Field 2 col  13 lth   6 >Second<
Field 3 col  22 lth   5 >Third<
$ 

If you want the exact original string starting from (say) the third field, use:

if (3 in col) Tail = substr (tx, col[3]);

Answer 3

If you're always going to use $1 as the key and the rest of the line (from $2) as the value, then you could use index to find where they are in $0:

1. Use index to get the location of $1 in $2
2. Then use the length of $1 to get the substring of $0 where the first position where $2 can be (for cases where $1 can contain a copy of $2)
3. Then use index again to get the location of $2, so that we can extract the substring starting at $2.

Example:

# foo.awk
function mysplit(array) {
    pos1 = index($0, $1)
    sub1 = substr($0, pos1 + length($1))
    pos2 = index(sub1, $2)
    sub2 = substr(sub1, pos2)
    array[$1] = sub2
}

{mysplit(arr)}

END {
    for (i in arr) {
        printf "[%s]:   |%s|\n", i, arr[i]
    }
}

With your examples:

% echo -n -e " \tFirst \t\t  Second \t Third  \t" | awk -f foo.awk -F " " | sed 's/\t/\\t/g'
[First]:   |Second \t Third  \t|
% echo -n -e " \tFirst \t\t  Second \t Third  \t" | awk -f foo.awk -F "\t" | sed 's/\t/\\t/g'
[ ]:   |First \t\t  Second \t Third  \t|
% echo -n -e " \tFirst \t\t  Second \t Third  \t" | awk -f foo.awk -F "[ \t]+" | sed 's/\t/\\t/g'
[]:   |First \t\t  Second \t Third  \t|