Parsing text file with nested delimiters on the same line

Question

I'd like to parse the btrfs scrub statistics that btrfs places in a file named /var/lib/btrfs/scrub.status.<uuid>. The file effectively consists of key/value pairs where key and value are separated by semicolons and tuples are separated by a pipe ("|") character:

scrub status:1
b5872f47-c87e-47ac-b036-4f2725cf6dc6:1|data_extents_scrubbed:4996799|tree_extents_scrubbed:1436139|data_bytes_scrubbed:156663988224|tree_bytes_scrubbed:23529701376|read_errors:0|csum_errors:0|verify_errors:0|no_csum:733729|csum_discards:0|super_errors:0|malloc_errors:0|uncorrectable_errors:0|corrected_errors:0|last_physical:202959224832|t_start:1597847400|t_resumed:0|duration:67|canceled:0|finished:1

In the end I'd like to emit each key/value pair on a separate line such as (this will serve as Prometheus metrics):

scrub status 1
data_extents_scrubbed 4996799
tree_extents_scrubbed 1436139
[...]

I have made various attempts to accomplish this nested parsing using sed/awk, but have failed so far.

Any hints how to accomplish this are much appreciated!

Answer 1

• Any sed:

  sed 's/:/ /g;s/|/\
  /g' file
  

• GNU sed:

  sed 's/:/ /g;s/|/\n/g' file
  

Here we change every : with a space and every | with a newline character.

With awk, you may set the record separator to the pipe character and then substitute : with space.

awk 'BEGIN{RS="|"}{gsub(":"," ");print}' file

Answer 2

With GNU awk for multi-char RS:

$ awk -v RS='[|]|\n$' -F':' '{$1=$1}1' file
scrub status 1
b5872f47-c87e-47ac-b036-4f2725cf6dc6 1
data_extents_scrubbed 4996799
tree_extents_scrubbed 1436139
data_bytes_scrubbed 156663988224
tree_bytes_scrubbed 23529701376
read_errors 0
csum_errors 0
verify_errors 0
no_csum 733729
csum_discards 0
super_errors 0
malloc_errors 0
uncorrectable_errors 0
corrected_errors 0
last_physical 202959224832
t_start 1597847400
t_resumed 0
duration 67
canceled 0
finished 1

Among other things, compared to an awk solution that just sets RS to |, the above won't print a spurious blank line at the end of its output.

Answer 3

Try:

$ awk -F'|' '{for (i=1;i<=NF;i++) {sub(/:/, " ", $i); print $i}}' file
scrub status 1
b5872f47-c87e-47ac-b036-4f2725cf6dc6 1
data_extents_scrubbed 4996799
tree_extents_scrubbed 1436139
data_bytes_scrubbed 156663988224
tree_bytes_scrubbed 23529701376

How it works:

• -F'|' sets the field separator to |.

• {for (i=1;i<=NF;i++) {sub(/:/, " ", $i); print $i}} loops through each field, substitutes a blank for : and prints the field.

Answer 4

sed

sed -e 'y/:|/ \n/' file

awk It's a matter of setting the record- and field-separators to the values shown and then have awk one more go at the field splitting operation. Finally, the records are unconditionally printed.

awk '
  BEGIN { RS = "|"; FS = ":" }
  {$1=$1}1
' file

Answer 5

Isn't a tr the fastest way to convert characters?

$ <file tr ':|' ' \n'

scrub status 1
b5872f47-c87e-47ac-b036-4f2725cf6dc6 1
data_extents_scrubbed 4996799
tree_extents_scrubbed 1436139
data_bytes_scrubbed 156663988224
tree_bytes_scrubbed 23529701376
read_errors 0
csum_errors 0
verify_errors 0
no_csum 733729
csum_discards 0
super_errors 0
malloc_errors 0
uncorrectable_errors 0
corrected_errors 0
last_physical 202959224832
t_start 1597847400
t_resumed 0
duration 67
canceled 0
finished 1

Of course, tr works only with one-byte characters. That in not a problem here for : and |.

An equivalent for sed (which may work with multi byte characters (GNU/ATT)) is:

<file sed 'y/:|/ \n/'