0

I'm trying to write a shell script that scans both columns 1 and 7 for user-specified values, with $3 being matched in 7th column and $2 being a year matched in first column (and $1 the filename). I can match the 7th col just fine, but keep receiving the error 'sed: -e expression #1, char 7: unknown command: `2' when attempting to add the variable to match in first column. I have to use sed instead of awk for this unfortunately. Here's what I have thus far:

sed -n "/"$2",[^,],[^,],[^,],[^,],[^,]*,"$3"/p" "$1"

what should I try to fix this? I'm quite a noobie so all help is welcome. Tried shellcheck but it hasn't helped. The script runs fine when scanning 7th column but adding the $2 aspect causes errors.

Edit: Sample invocation:

./script.sh filename "2015.2016" "South"

Sample file (with "..." being a column with text inside):

2021/2022,text,text,text,text,text,South,text,...,...
2021/2022,text,text,text,text,text,North,text,...,...
2015/2016,text,text,text,text,text,South,text,...,...
2014/2015,text,text,text,text,text,West,text,...,...

Intended output:

2015/2016,text,text,text,text,text,South,text,...,...

Edit: While I agree that it's indeed not the best tool for the job, I figured out how to accomplish this with sed after playing around a bit:

#!/bin/bash

year="$2"
name="$3"

sed -n '/^'"$year"',\([^,]*,\)\{5\}'"$name"'\(,.*\)\{0,\}$/p' "$1" 

Dunno if I'll ever make use of it but maybe it will help someone else out. Note that the input needs a "." rather than "/" in order to yield accurate result.

6
  • 1
    We can't really help unless you tell us exactly what the variables hold, how their values are set and also give us an example input and expected output so we can test our answers. It is a bit odd to be using sed for this, awk is easier for field-separated data, and if this is an actual csv file, then you really should use a dedicated csv parser since csv fields can contain newlines and comas.
    – terdon
    Jun 18, 2022 at 22:59
  • That makes sense! let me edit
    – AstralV
    Jun 18, 2022 at 23:02
  • sed is the wrong tool for processing CSV files/data. Use awk. Or perl (optionally with the Text::CSV module) for this. Or better yet, miller or csvkit
    – cas
    Jun 19, 2022 at 5:08
  • Please post your answer as an answer instead of an edit to the question. Also, if you still want improved answers, even only using sed, you really need to give us an example input and expected output to play with as I asked yesterday. Finally, note that you can put your whole sed in double quotes, there's no reason to switch between single and double: sed -n "/^$year,\([^,]*,\)\{5\}$name\(,.*\)\{0,\}$/p" "$1"
    – terdon
    Jun 19, 2022 at 9:35
  • What you have added to your question as a solution will fail in various, sometimes cryptic, ways when the user of your tool calls it as script.sh file '2.*' 'South' or script.sh '2020' 'S/N' or any of many other strings. That's why the script in my answer that uses sed is necessarily more complicated than yours, to protect against any of that.
    – Ed Morton
    Jun 19, 2022 at 12:04

1 Answer 1

3

sed isn't a good tool for this, just use awk:

$ cat script.sh
#!/usr/bin/env bash

awk -F',' '$1==a && $7==b' a="$2" b="$3" "$1"

$ ./script.sh file.csv "2021" "South"
2021,text,text,text,text,text,South,text

Since you need a sed solution, though, using a POSIX sed:

$ cat script.sh
#!/usr/bin/env bash

a=$(sed 's/[^^\\]/[&]/g; s/\^/\\^/g; s/\\/\\\\/g' <<< "$2")
b=$(sed 's/[^^\\]/[&]/g; s/\^/\\^/g; s/\\/\\\\/g' <<< "$3")
sed -n '/^'"$a"',\([^,]*,\)\{5\}'"$b"'\(,.*\)\{0,\}$/p' "$1"

$ ./script.sh file.csv "2021" "South"
2021,text,text,text,text,text,South,text

See is-it-possible-to-escape-regex-metacharacters-reliably-with-sed for why the first 2 sed commands are necessary but in short it's because you need to do a literal string match while sed only supports regexp matches, not literal string matches, so you have to escape every possible regexp metachar (actually you have to escape every char as any char could be the sed script delimiter, e.g. /) to make it act like a literal char. That is the 2nd big clue that sed is the wrong tool for the job (the first being that it doesn't split input into fields) vs a tool such as awk that simply supports literal string matches.

11
  • Unfortunately I have to use sed for this :(
    – AstralV
    Jun 18, 2022 at 23:13
  • 2
    If this is homework, it's a bad lesson in using the wrong tool for a job as it'd take a lot of work to create a sed script that can do this job robustly since, among other issues, sed doesn't understand literal strings, it only can search for regexp matches, so you'd have to escape any regexp characters that the user inputs which is non-trivial (see stackoverflow.com/q/29613304/1745001) and then you have to worry about delimiters, etc. It's just a bad idea.
    – Ed Morton
    Jun 18, 2022 at 23:14
  • Trust me, I agree lol
    – AstralV
    Jun 18, 2022 at 23:17
  • 4
    Give your teacher an F :-).
    – Ed Morton
    Jun 18, 2022 at 23:17
  • @AstralV I just added the way to do it using a POSIX sed.
    – Ed Morton
    Jun 18, 2022 at 23:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.