I have a script which uses sed in a loop to delete lines containing words from a list:

for i in $list; do
  sed -i "/$i/d" file

$list is created from an SQL query:

list=$(psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC")

The problem is my company recently implemented tighter auditd rules which include all fchown or fchmod syscalls. Using strace, I see every sed -i includes one fchown and fchmod syscall.

With a large list variable, iowait skyrockets and the system actually bogs down because auditd is writing a ridiculous amount of lines to its log.

I'm trying to figure out if there's a way to pass the list variable to sed in one line so the file is only opened and closed once.

I've already asked, any changes to the auditd rules is a non-starter.


  • 2
    What's $list ? Is it a list of words saved in a file, one per line or an array or what ? Jan 23 '19 at 17:12
  • $list is one word per line. Jan 23 '19 at 17:15
  • From where do you get list? If all the words or patterns are in a file line by line you could use fgrep -v -f patternfile inputfile > file && mv tmpfile file of grep ... if you have regex patterns.
    – Bodo
    Jan 23 '19 at 17:22
  • You could try to convert your list in a string in that way, that the final sed command that is exceuted looks like this: sed -i -E "/(word1|word2|word3)/d" file
    – finswimmer
    Jan 23 '19 at 17:25
  • I added how the list is created to the post. I was wrong earlier, it's not one word per line, rather words separated by a single space. Jan 23 '19 at 17:35

By only running sed once and using a single sed script rather than invoking a zillion in-line commands one-by-one:

for i in $list; do
  echo "/$i/d" >> "$sedscript"
sed -f "$sedscript" -i /path/to/file
rm -f "$sedscript"
  • I'm liking this solution, but I'm getting sed: can't find label for jump to 'mp.Vbqh8BAFGv' The 't' is getting dropped from the mktemp filename somewhere. Jan 23 '19 at 17:50
  • This works without creating a temp file: sedscript=$(for i in $list; do echo "/$i/d"; done) && sed -e "$sedscript" -i /path/to/file' Jan 23 '19 at 18:15
  • My mistake; sed -e should have been sed -f; -e is "run this command right here" (which is implied by default), while -f is "run the commands in this file".
    – DopeGhoti
    Jan 23 '19 at 18:19

If you can carefully assign the words in list, you could use bash parameter expansion to replace every space with the regex alternator |, then ask grep to do the work for you:

list='auditd the to'
cp file temp &&  
grep -Ev "${list// /|}" temp > file && 
rm temp 
  • This relies on the assumption (also present in your sed example) that none of the "list" values contain regular expression tokens (that you didn't expect).
    – Jeff Schaller
    Jan 23 '19 at 17:27

If your shell supports arrays, you could do something like

mapfile -t arr < <(psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC")

to put the output of the command into a shell array, then

printf '/%s/d\n' "${arr[@]}" | sed -i.bak -f - somefile

to apply the sequence of deletions to somefile, in place. The usual caveats apply if the fields returned from your psql command might contain characters that are special to sed.

This is similar to @DopeGhoti's solution, except that it avoids the temporary script file (and should handle whitespace in table fields, should that be necessary).


If this: psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC" gives you a white-space delimited list of words, try this:

$ psql -d dbname -t -c "SELECT foo from table ORDER BY column DESC" \
| awk -v FS=" " -v OFS="|" '{$1=$1; print "("$0")"}' \
| xargs -I {} sed -i -E '/{}/d' file

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.