5

From within a bash script, how can I use sed to write to a file where the filename is stored in a bash variable? Output redirection won't do because I want to edit one file in place, pulling lines that match a regex into a different file and deleting them in the first file.

Something like:

sed -i '/^my regex here$/{;w file2;d;}' file1

...but where file1 and file2 are both actually variables that hold the filenames. (Such as file2=$(mktemp).)

So what I really want is variable expansion for the filename, but no expansion for the rest of the sed command, and leaving the whole thing as a single argument passed to the sed command.

For whatever reason, the following does not work:

sed -i '/my regex here/{;w '"$file2"';d;}' $file1

It says "unmatched {" and I can't see why.

Any way I can do this?

3
  • I don't get your sed, however, simply using $file1 will substitute a filename if file1 is a variable... Maybe that's not what you mean.
    – Brian
    Nov 3, 2015 at 0:35
  • I noticed later in my /tmp directory that I had a file named /tmp/tmp.(blahblah);d;} so ... maybe I need a space after the filename? Or does it have to be a newline?
    – Wildcard
    Nov 3, 2015 at 0:55
  • The variables are not the problem. Your syntax seems to be wrong in general. Try it with actual file names instead of variables and you'll get the same error. As far as I can tell, the w doesn't like being inside {}. For example, this works (but doesn't delete the line): sed -i '/regex/w file' but this fails with the error you show: sed '/regex/{w file}'
    – terdon
    Nov 3, 2015 at 0:58

3 Answers 3

4

Because you're using a single sed expression, everything that follows after the w (including the }) is interpreted as the wfile name:

The argument wfile shall terminate the editing command.

You can see that if you add a second command } e.g. like:

sed -e '/my regex here/{w '"$file2"';d;}' -e '}' $file1

then the lines matching my regex here will be saved in a file named whatever;d;} where whatever is whatever $file1 expands to.
The correct syntax is via separate commands, either with several expressions:

sed -e '/my regex here/{w '"$file1" -e 'd' -e '}' $file2

or one command per line:

sed '/my regex here/{
w '"$file1"'
d
}
' $file2
4
  • Nice! Thanks! :) (Yes, we got it both right at the same time!) I accidentally switched file1 and file2 in my second command in the question; fixed now. You may want to fix it here as well. Also, the reason I put only the filename in double quotes in my answer, is that a sed regex can often include $ which will then cause very weird errors if in double quotes.
    – Wildcard
    Nov 3, 2015 at 2:02
  • Why don't you use double quotes for the entire 1st command instead of the ungainly ' " construct? And why Something like sed -e "/regex/{w $file" -e 'd' -e '}' file. Also, any reason why you don't combine the d and } into one command? That works on my sed, is that a GNUism? I mean something like sed -e "/regex/{w $file" -e 'd}'.
    – terdon
    Nov 3, 2015 at 11:19
  • @terdon, if you see my answer, I realized that -e "w $filename" is the cleanest answer. Everything before and after that can be in a separate -e argument in single quotes for reliability. (It's not unusual for sed commands to include a $, either in an address or a substitute command.) And the w doesn't need to be in single quotes, because it will always be that exact fixed string, which is unaffected by double quotes.
    – Wildcard
    Nov 4, 2015 at 1:55
  • I always use the quoting method which will allow the least expansion. If you don't need any expansion, use single quotes (unless you need a single quote in your string); if you don't need anything but variable expansion, use double quotes. So I "add expansion functionality" to my scripts as needed by removing quotes, rather than tracing down crazy bugs and fixing them by adding quotes.
    – Wildcard
    Nov 4, 2015 at 1:57
4

I found the answer on Stackoverflow under "How do I let sed 'w' command know where the filename ends?"

As terdon pointed out, the issue is not the variable—but it has nothing to do with curly brackets, either; try sed '/^l/w testing;p' and you will see it doesn't throw any error, but writes all the lines starting with an l into a file named testing;p.

The issue is actually the lack of a newline after the filename.

So the answer is to either use an inline newline in the sed command:

sed -i '/my regex here/{;w '"$file2"'
d;}' $file1

Or, what is much cleaner, use two separate -e arguments:

sed -i -e '/my regex here/{;w '"$file2" -e 'd;}' $file1

If you don't like the two adjacent quotes (which you might easily slip on), just put the write command in its own argument:

sed -i -e '/^my regex here$/{' -e "w $file2" -e 'd;}' $file1
3
  • 1
    Similar behavior to this on the test t and branch b commands, the branch label requires a newline or end of string to terminate the label.
    – RobertL
    Nov 3, 2015 at 2:55
  • @RobertL - they don't always require a newline/separate command, not with gnu sed at least, whereas r and w always do for the reason explained in my post. Nov 3, 2015 at 7:38
  • 1
    @don_crissti Yes. Forgot about gnu. I think it's always required on BSD sed but too tired to test again right now.
    – RobertL
    Nov 3, 2015 at 7:47
-1

Use the double quotes instead the single ones in to quote the sed command. And don't forget to escape suitable characters.

More info about quoting variables in bash in this link

1
  • 1
    The issue is not the variable, the syntax itself is wrong. Try it.
    – terdon
    Nov 3, 2015 at 0:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.