0

Most of the time, I am given an input file that looks like this:

java-1.8.0-openjdk.x86_64  1:1.8.0.232.b09-1.el6_10        asyum:ol6_latest
java-1.8.0-openjdk-headless.x86_64 1:1.8.0.232.b09-1.el6_10        asyum:ol6_latest
kernel.x86_64              2.6.32-754.23.1.el6             asyum:ol6_latest

...and I have a while loop that is able to successfully capture field1 and field2 from each line using this code:

f1=$(echo $line | awk '{print $1}')
f2=$(echo $line | awk '{print $2}')

However, I'll occasionally get input files that look similar to this:

java-1.8.0-openjdk.x86_64  1:1.8.0.232.b09-1.el6_10        asyum:ol6_latest
java-1.8.0-openjdk-headless.x86_64
                           1:1.8.0.232.b09-1.el6_10        asyum:ol6_latest
kernel.x86_64              2.6.32-754.23.1.el6             asyum:ol6_latest

(Notice there is an extra line feed that splits line 2.)

Do you have any tips/tricks for handling this type of input programmatically (instead of manually cleaning up the input file)?

5
  • Ah, my bad! I had the wrong understanding regarding what a CR is until I read your reply. I meant LF. (My input files do not contain any CRs.)
    – user26241
    Jan 16, 2020 at 20:59
  • Actually, yes--leading spaces after the extra LF are always there. (Never realized this before.) Not sure if this matters, but sometimes the extra LF is after the SECOND field, like this:
    – user26241
    Jan 16, 2020 at 21:14
  • line 1: java-1.7.0-openjdk.x86_64 1:1.7.0.241-2.6.20.0.0.1.el7_7
    – user26241
    Jan 16, 2020 at 21:15
  • line 2: asyum:ol7_latest
    – user26241
    Jan 16, 2020 at 21:15
  • (FYI, line 2 DOES have preceding spaces before "asyum"; I just can't get them to show up here.)
    – user26241
    Jan 16, 2020 at 21:16

5 Answers 5

1

This is a solution that relies solely on leading space(s). It doesn't rely on expected number of fields.

The manual of GNU sed provides a way to "join lines that start with whitespace":

sed -E ':a ; $!N ; s/\n\s+/ / ; ta ; P ; D'

The manual claims a portable (non-GNU) variation is:

sed -e :a -e '$!N;s/\n  */ /;ta' -e 'P;D'
0

This should achieve what's required:

#!/usr/bin/env bash

while read line; do
    declare -a array=($line)
    if  test ${#array[@]} != 3; then
        read line2
        array+=($line2)
    fi
    f1=${array[0]}
    f2=${array[1]}
    echo $f1 $f2
done

run with : test.sh < inputfile

It checks if a line contains three items, if not, join next line.

5
  • You set up an array beautifully but then you revert to awk to pick out field values from the line that are almost already in your array. What!? Jan 16, 2020 at 21:45
  • In case the line is incomplete, the array does not contain everything.
    – Philippe
    Jan 16, 2020 at 21:47
  • thank you Philippe! This worked!
    – user26241
    Jan 16, 2020 at 22:01
  • Instead of line+=" $line2" you put array+=($line2) and then you have "${array[0]}" as the first element and "${array[1]}" as the second. Jan 16, 2020 at 23:32
  • You are right, updated.
    – Philippe
    Jan 16, 2020 at 23:38
0

As you are using awk anyhow, why not rely on it entirely? Try

read f1 f2 <<< $(awk '{while (NF < 3) {getline X; $0 = $0 FS X};  print $1, $2}' file)
0

You could use xargs -n3 -a inputfile to rearrange the input file and put three elements on each line.

while read -r f1 f2 temp; do
  echo "f1=$f1"
  echo "f2=$f2"
done < <(xargs -n3 -a inputfile)
0

Under the assumption that every well-formed line of your input data was made of three words (loosely defined here as strings not containing any blank character), you may preprocess your input data:

<input_file tr -s '[:blank:]' '\n' | paste - - - | column -t

Here, we are replacing any blank character (including <newline> characters) with a single <newline>, rearranging the result into lines of three words and nicely formatting the output—the last one being an unnecessary, cosmetic step, given your question.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .