I frequently find myself needing to mangle text from one format to another. A typical case is I have a log file that contains some information, and I want to fish out a subset of that information, formatted in a very specific way.
Now, if each "record" is on the same line, the task is relatively easy:
- Optionally use
grepto match only interesting lines.
sedto match the fields of interest, capture them, and output the result formatted as required.
The only slight hiccup is the need to make a regex that matches the entire line, so
sed will discard it and output only the desired output. But that's not too hard.
If the field capture problem is simple enough,
cut might be sufficient, or perhaps
Now, the real problem: What if the fields I want span multiple lines?
Suppose, for example, somebody does
stat * >LOG.TXT
Well, now we have a file
LOG.TXT that contains all the info we want... but it looks like
File: 'src.7z' Size: 269430 Blocks: 528 IO Block: 4096 regular file Device: 801h/2049d Inode: 799155 Links: 1 Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root) Access: 2020-07-02 09:01:09.269292914 +0100 Modify: 2020-07-02 09:00:53.237293629 +0100 Change: 2020-07-02 09:00:53.237293629 +0100 Birth: -
And perhaps we want to generate something that looks more like
VerifyFile("src.7z", 269430, 0644);
Now, the correct way to do this is to just ask
stat to output it that way in the first place! In general, it's always better to have the source program generate the formatting you want to start with, rather than trying to Frankenstein the text afterwards. Sadly, life is not always so simple. Perhaps you have a log file were you want to pair connection events with disconnection events, or a file listing where you want to pair each
.c file with its matching
.o file, or whatever. For whatever reason, you end up with a text mangling task.
So what's the best way to approach this? How do you gather fields from multiple separate lines, and munge them onto the same line, but still keep records separate? (I.e., don't put everything onto one giant line, only the fields related to the same record.)
I don't have a general method for doing this type of task. Some thoughts:
- You can use
grepto extract only the part that matches a regex, and thereby extract the fields you want. However, I don't see a way to extract all the interesting fields without running
grepmultiple times, resulting in destroying the relative ordering between fields.
sedmakes it more awkward to extract just the matching bits, but does allow to keep the fields in the same relative order. Now you still have to somehow gather fields onto the same line.
awkhas the concept of fields and records. I believe it can do regex matching. (I don't really know
awk.) Maybe that's the way to approach this? It seems quite complicated though.
- Can you do something with Python? That usually seems to be installed. I don't know if it has any regex capabilities though.
Usually I eventually come up with something that kinda works, but it's never very readable or maintainable. I'm hoping there's an easier way.