This sed syntax won't remove blank lines

Question

I have this input file:

...
https://long-url/something?value1=999&something=xyz?value2=555
https://long-url/something?value1=534&something=xyz?value2=2312


https://long-url/something?value1=534&something=xyz?value2=233
https://long-url/something?value1=534
...

I want to capture the value1 and value2 and output them as csv, and drops any lines that don't match:

...
999,555
534,2312
534,233
...

I have no problem on capturing the values, but I'm having issue removing any unwanted lines:

sed 's/.*categoryid=\([0-9]*\).*courseid=\([0-9]*\).*/\1,\2/g;/^[^0-9]\+/d' input-file

Basically, the /^[^0-9]\+/d is not working as intended. I was thinking that it would remove any lines that don't start with any digit, but instead, it gave me:

999,555
534,2312


534,233

---

My sed version: sed (GNU sed) 4.7

Answer 1

You are using a substitute, followed by a delete:

sed -e 's/.../.../g' -e '/.../d'

That's a bit volatile, because your /.../d might accidentally match something else in the file's contents.

Instead, I would only print the lines that were affected by the substitute. That can be done with -n (to avoid printing by default), and then adding p to the end of the line to print matched lines. That avoids the need for a second sed statement.

sed -ne 's/.../.../gp'

I would also remove the trailing g since you don't need more than one match per line:

sed -ne 's/.../.../p'

The result is:

sed -ne 's/.*categoryid=\([0-9]*\).*courseid=\([0-9]*\).*/\1,\2/p' input-file

---

In action:

$ sed -ne 's/.*value1=\([0-9]*\).*value2=\([0-9]*\).*/\1,\2/p' << EOF
https://long-url/something?value1=999&something=xyz?value2=555
https://long-url/something?value1=534&something=xyz?value2=2312


https://long-url/something?value1=534&something=xyz?value2=233
https://long-url/something?value1=534
EOF

outputs

999,555
534,2312
534,233

Answer 2

Your expression, ^[^0-9]\+, forces the matching of at least one non-digit, which means empty lines won't match it, as they don't contain non-digits.

Personally, I would do this the following way:

sed -e '/.*categoryid=\([0-9]*\).*courseid=\([0-9]*\).*/!d' \
    -e 's//\1,\2/' input-file

This first deletes all lines that do not match your specific regular expression (for example, empty lines). The second expression performs the substitution. When an empty pattern is used (as in the substitution in the second expression), the most recently used pattern is re-used.

Note that you don't need /g at the end of the substitution command as you only ever expect to match the pattern once per input line.

Answer 3

To match /^[^0-9]\+/, the line must must contain at least one character, i.e. it can't be empty.

Just don't remove the lines that start with a digit:

/^[0-9]/!d