gdrive has a sub command list which prints a list of files like the following example:

gdrive list


Id                                  Name                      Type   Size     Created
1sV3_a1ySV0-jbLxhA8NIEts1KU_aWa-5   info.pdf                  bin    10.0 B   2018-08-27 20:26:20
1h-j3B5OLryp6HkeyTsd9PJaAtKK_GYyl   2018-12-ss-scalettapass   dir             2018-08-27 20:26:19

I'm trying to parse this output using tools like awk and sed without success.

The problems are empty 'fields' in the size column and the dynamic widths of the columns.

Has anybody an idea how to parse this output?

  • 2
    it appears to me that the columns are tab-delimited; can you confirm with gdrive list | sed -n l? – Jeff Schaller Aug 27 '18 at 19:58
  • @JeffSchaller: I also looked at this code. These tabs are somehow converted to spaces. Besides gdrive list | sed -n l I also tried gdrive list | cat -A, which did not make any special characters visible. – Ray Aug 28 '18 at 17:44

awk can deal with fixed width data. First we need to determine the column widths:

fieldwidths=$(head -n 1 file | grep -Po '\S+\s*' | awk '{printf "%d ", length($0)}')

This value is "36 26 7 9 7 " -- the last field is larger than 7 chars. Let's arbitrarily make it 70 chars:

fieldwidths=${fieldwidths/% /0}

Now, let's read the data and turn it into CSV:

awk -v FIELDWIDTHS="$fieldwidths" '{
    for (i=1; i<=NF; i++) {
        val = $i
        sub(/ *$/, "", val)
        gsub(/"/, "\"\"", val)
        printf "%s\"%s\"", (i==1 ? "" : ","), val
    print ""
}' file


"1sV3_a1ySV0-jbLxhA8NIEts1KU_aWa-5","info.pdf","bin","10.0 B","2018-08-27 20:26:20"
"1h-j3B5OLryp6HkeyTsd9PJaAtKK_GYyl","2018-12-ss-scalettapass","dir","","2018-08-27 20:26:19"

The same functionality with perl

perl -lne '
    if ($. == 1) {
        @head = ( /(\S+\s*)/g );
        pop @head;
        $patt = "^";
        $patt .= "(.{" . length($_) . "})" for @head;
        $patt .= "(.*)\$";
    print join ",", map {s/"/""/g; s/\s+$//; qq("$_")} (/$patt/o);
' file
  • Not all awk versions offer the FIELDWIDTHS option. – RudiC Aug 28 '18 at 8:24
  • @glenn-jackman: Thank you for your answers. Finally, I chose your Perl-based solution that was most understandable to me. – Ray Aug 28 '18 at 17:54

You can do this with Perl using the unpack function by creating the unpacking template dynamically by examining the header (1st line):

perl -lpe '
    $fmt //= join "", map("A" . length(), /\H+\h+(?=\H)/g), "A*";
    $_ = join ",", map { s/"/""/gr =~ s/(.*)/"$1"/r } unpack $fmt;
' input-file.txt


  • -p will make perl consume the file on a per-line basis. Each line, aka, record, is referred to as $_. Another effect of -p is it autoprints the current record before going to fetch the next one.
  • -l does 2 things, sets ORS = RS = \n
  • The regex /\H+\h+(?=\H)/g shall fetch all the fields except the last and then these are fed to map.
  • map computes the lengths of these fields and prefixes an "A" to each.
  • In lieu of not selecting the last field above, we add a catch-all "A*".
  • These then are passed to join which sticks them together into a string using the null delimiter. So the unpack format is ready for use and is not computed again because of the //= operator which is the defined-or function.
  • Now, armed with the dynamically created unpack format, we proceed to apply to every line, including the header.
  • unpack unpacks a string, in our case the current line, using the format provided and emits fields unpacked.
  • These emitted fields are then input to map which operates on each one-by-one, and performs the steps outlined in the { ... } code. In our case, in each field we do the following: a) double the double quotes. b) enrobe the field with double quotes.
  • After map is done editing the fields, it throws them over to join, which joins them using the comma , to form a nice little CSV file.
  • P.S.: Notice that we didn't have to trim the trailing blanks in the fields generated by unpack, coz, unpack does that for you when using the A (A for ASCII) formatting character.


"1sV3_a1ySV0-jbLxhA8NIEts1KU_aWa-5","info.pdf","bin","10.0 B","2018-08-27 20:26:20"
"1h-j3B5OLryp6HkeyTsd9PJaAtKK_GYyl","2018-12-ss-scalettapass","dir","","2018-08-27 20:26:19"

This can be done by the sed tool, but would need a two-pass approach, in that first, using the header line of the input, we generate a sed script dynamically, which then operates upon the input file (including the header as well) to perform the desired operation, as shown:

cmd=$(< "$if" head -n 1 | perl -lne 'print join $/, reverse map { $s += length();qq[s/./\\n/$s] } /\H+\h+(?=\H)/g')
sed -e '
' < "$if"
  • Just a heads-up, Rakesh -- you appear to have two unregistered accounts, here and here – Jeff Schaller Aug 28 '18 at 17:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.