Extract data fields from multiple lines

Question

I frequently find myself needing to mangle text from one format to another. A typical case is I have a log file that contains some information, and I want to fish out a subset of that information, formatted in a very specific way.

---

Now, if each "record" is on the same line, the task is relatively easy:

• Optionally use grep to match only interesting lines.
• Use sed to match the fields of interest, capture them, and output the result formatted as required.

The only slight hiccup is the need to make a regex that matches the entire line, so sed will discard it and output only the desired output. But that's not too hard.

If the field capture problem is simple enough, cut might be sufficient, or perhaps awk.

---

Now, the real problem: What if the fields I want span multiple lines?

Suppose, for example, somebody does

stat * >LOG.TXT

Well, now we have a file LOG.TXT that contains all the info we want... but it looks like

  File: 'src.7z'
  Size: 269430          Blocks: 528        IO Block: 4096   regular file
Device: 801h/2049d      Inode: 799155      Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2020-07-02 09:01:09.269292914 +0100
Modify: 2020-07-02 09:00:53.237293629 +0100
Change: 2020-07-02 09:00:53.237293629 +0100
 Birth: -

And perhaps we want to generate something that looks more like

VerifyFile("src.7z", 269430, 0644);

Now, the correct way to do this is to just ask stat to output it that way in the first place! In general, it's always better to have the source program generate the formatting you want to start with, rather than trying to Frankenstein the text afterwards. Sadly, life is not always so simple. Perhaps you have a log file were you want to pair connection events with disconnection events, or a file listing where you want to pair each .c file with its matching .o file, or whatever. For whatever reason, you end up with a text mangling task.

So what's the best way to approach this? How do you gather fields from multiple separate lines, and munge them onto the same line, but still keep records separate? (I.e., don't put everything onto one giant line, only the fields related to the same record.)

I don't have a general method for doing this type of task. Some thoughts:

• You can use grep to extract only the part that matches a regex, and thereby extract the fields you want. However, I don't see a way to extract all the interesting fields without running grep multiple times, resulting in destroying the relative ordering between fields.
• sed makes it more awkward to extract just the matching bits, but does allow to keep the fields in the same relative order. Now you still have to somehow gather fields onto the same line.
• awk has the concept of fields and records. I believe it can do regex matching. (I don't really know awk.) Maybe that's the way to approach this? It seems quite complicated though.
• Can you do something with Python? That usually seems to be installed. I don't know if it has any regex capabilities though.

Usually I eventually come up with something that kinda works, but it's never very readable or maintainable. I'm hoping there's an easier way.

Answer 1

Both awk and python can be a good fit here. They both support associative arrays (or dictionaries) and both have regex built-in.

There's perhaps some generic way to approach them, but I tend to optimize based on actual sample input/output that I can play with. Here's your stat problem solved with awk.

$ cat kv.awk 
/File:/ {
    f = $2
}

/Size:/ {
    info[f] = $2
}

/Access.*Uid/ {
    gsub(/\(|\/.*/, "", $2)
    info[f] = info[f] OFS $2
}

END {
    for (k in info) {
        print "VerifyFile(\"" k "\"", info[k] ");"
    }
}

The file name is used as key to keep all things associated with it together. At the end, you can display the results in whatever format you need. GNU awk supports multidimensional arrays too.

Here's a sample run:

$ stat ip.txt test.txt > file.txt

$ awk -v OFS=', ' -f kv.awk file.txt
VerifyFile("'test.txt'", 254, 0664);
VerifyFile("'ip.txt'", 301, 0664);

To learn awk, I'd recommend https://www.gnu.org/software/gawk/manual/gawk.html which covers GNU awk as well as mentions details about other implementations.